2023-10-30T11:54:00.490 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-30T11:54:00.491 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --verbose --archive /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076 -- /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076/orig.config.yaml 2023-10-30T11:54:00.533 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T11:54:00.613 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7441076' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: d0db9f2f95b9d6344950185071cace2a50d296eb owner: scheduled_amathuri@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 1876 sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d sleep_before_teardown: 0 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d0db9f2f95b9d6344950185071cace2a50d296eb targets: smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErD+fKORMbAS/4SGdN1qk+fU5KuhzMeu5Q6Ym1h8GMQsH5+45LOEyN9dgz8ltlxPMWIuTBvuay8FM/2eK7wFqM= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKgJDD52JQV1IuyEaPL+QAJj7lMW3qMUl3NiI2iKvVkDuf8z8E/576IK4HXp620zEqzKzeEFUb7IQIoHyd7zmSY= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-30_10:36:20 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T11:54:00.614 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa; will attempt to use it 2023-10-30T11:54:00.615 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks 2023-10-30T11:54:00.615 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T11:54:00.617 INFO:teuthology.task.internal:Checking packages... 2023-10-30T11:54:00.640 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'cc36dd85d142c6fdb77515cad8af108cec98a86d' 2023-10-30T11:54:00.640 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T11:54:00.641 INFO:teuthology.packaging:ref: None 2023-10-30T11:54:00.641 INFO:teuthology.packaging:tag: None 2023-10-30T11:54:00.641 INFO:teuthology.packaging:branch: main 2023-10-30T11:54:00.642 INFO:teuthology.packaging:sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T11:54:00.642 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2023-10-30T11:54:00.975 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6863.g636384c4 2023-10-30T11:54:00.976 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T11:54:00.983 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T11:54:00.983 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T11:54:00.998 INFO:teuthology.task.internal:Saving configuration 2023-10-30T11:54:01.013 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T11:54:01.020 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T11:54:01.046 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:48:37.669932', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErD+fKORMbAS/4SGdN1qk+fU5KuhzMeu5Q6Ym1h8GMQsH5+45LOEyN9dgz8ltlxPMWIuTBvuay8FM/2eK7wFqM='} 2023-10-30T11:54:01.071 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:48:37.667921', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKgJDD52JQV1IuyEaPL+QAJj7lMW3qMUl3NiI2iKvVkDuf8z8E/576IK4HXp620zEqzKzeEFUb7IQIoHyd7zmSY='} 2023-10-30T11:54:01.072 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T11:54:01.078 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-10-30T11:54:01.079 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-10-30T11:54:01.079 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T11:54:01.150 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd044a6da60>, signals=[15]) 2023-10-30T11:54:01.152 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T11:54:01.157 INFO:teuthology.task.internal:Opening connections... 2023-10-30T11:54:01.158 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T11:54:01.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:54:01.245 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T11:54:01.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:54:01.329 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T11:54:01.336 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2023-10-30T11:54:01.355 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2023-10-30T11:54:01.355 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2023-10-30T11:54:01.412 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2023-10-30T11:54:01.413 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="9" 2023-10-30T11:54:01.413 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2023-10-30T11:54:01.413 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2023-10-30T11:54:01.414 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="9" 2023-10-30T11:54:01.414 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el9" 2023-10-30T11:54:01.414 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-30T11:54:01.414 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2023-10-30T11:54:01.414 INFO:teuthology.orchestra.run.smithi148.stdout:LOGO="fedora-logo-icon" 2023-10-30T11:54:01.415 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-30T11:54:01.415 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2023-10-30T11:54:01.415 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-30T11:54:01.415 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-30T11:54:01.415 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-30T11:54:01.416 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2023-10-30T11:54:01.442 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2023-10-30T11:54:01.460 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-10-30T11:54:01.460 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2023-10-30T11:54:01.517 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2023-10-30T11:54:01.518 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2023-10-30T11:54:01.518 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2023-10-30T11:54:01.518 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2023-10-30T11:54:01.518 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2023-10-30T11:54:01.519 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2023-10-30T11:54:01.519 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-30T11:54:01.519 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2023-10-30T11:54:01.519 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2023-10-30T11:54:01.519 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-30T11:54:01.520 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2023-10-30T11:54:01.520 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-30T11:54:01.520 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-30T11:54:01.520 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-30T11:54:01.521 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-10-30T11:54:01.539 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T11:54:01.551 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T11:54:01.557 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T11:54:01.558 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2023-10-30T11:54:01.561 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2023-10-30T11:54:01.576 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T11:54:01.582 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T11:54:01.582 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2023-10-30T11:54:01.618 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2023-10-30T11:54:01.747 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T11:54:01.799 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T11:54:01.814 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-30T11:54:01.814 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-30T11:54:01.815 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T11:54:01.816 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T11:54:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:54:01.833 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2023-10-30T11:54:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:54:01.834 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-10-30T11:54:01.890 INFO:teuthology.orchestra.run.smithi148.stdout:5.14.0-361.el9.x86_64 2023-10-30T11:54:01.890 INFO:teuthology.task.kernel:Running kernel on smithi148: 5.14.0-361.el9.x86_64 2023-10-30T11:54:01.891 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2023-10-30T11:54:01.892 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-361.el9.x86_64 2023-10-30T11:54:01.892 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-361.el9.x86_64 2023-10-30T11:54:01.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2023-10-30T11:54:03.900 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 16 MB/s | 6.3 MB 00:00 2023-10-30T11:54:03.906 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-9 - AppStream 16 MB/s | 6.3 MB 00:00 2023-10-30T11:54:05.233 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2023-10-30T11:54:05.517 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2023-10-30T11:54:06.465 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 7.9 MB 00:00 2023-10-30T11:54:08.390 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 7.9 MB 00:02 2023-10-30T11:54:09.054 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 11 MB/s | 18 MB 00:01 2023-10-30T11:54:10.250 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - AppStream 21 MB/s | 18 MB 00:00 2023-10-30T11:54:13.426 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 5.7 MB 00:01 2023-10-30T11:54:14.860 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 15 kB 00:00 2023-10-30T11:54:15.483 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.7 MB 00:01 2023-10-30T11:54:15.870 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 21 MB/s | 19 MB 00:00 2023-10-30T11:54:16.784 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 15 kB 00:00 2023-10-30T11:54:17.794 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 22 MB/s | 19 MB 00:00 2023-10-30T11:54:20.344 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2023-10-30T11:54:22.352 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2023-10-30T11:54:22.409 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-30T11:54:22.410 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-30T11:54:22.489 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T11:54:22.496 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T11:54:22.496 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-10-30T11:54:22.496 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T11:54:22.496 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T11:54:22.497 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-30T11:54:22.497 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-30T11:54:22.497 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-30T11:54:22.497 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-30T11:54:22.497 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-30T11:54:22.498 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T11:54:22.498 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T11:54:22.498 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T11:54:22.498 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2023-10-30T11:54:22.498 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T11:54:22.499 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 95 M 2023-10-30T11:54:22.499 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 120 M 2023-10-30T11:54:22.500 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T11:54:22.937 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 14 MB/s | 5.3 MB 00:00 2023-10-30T11:54:23.813 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 16 MB/s | 20 MB 00:01 2023-10-30T11:54:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): kernel-modules-core-5.14.0-378.el9.x86_6 22 MB/s | 32 MB 00:01 2023-10-30T11:54:24.417 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-30T11:54:24.417 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-30T11:54:24.515 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T11:54:24.525 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T11:54:24.525 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-10-30T11:54:24.525 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T11:54:24.526 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T11:54:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-30T11:54:24.526 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-10-30T11:54:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-30T11:54:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-30T11:54:24.527 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-30T11:54:24.527 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T11:54:24.527 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T11:54:24.527 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T11:54:24.527 INFO:teuthology.orchestra.run.smithi148.stdout:Install 4 Packages 2023-10-30T11:54:24.528 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T11:54:24.528 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 95 M 2023-10-30T11:54:24.529 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 120 M 2023-10-30T11:54:24.529 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T11:54:24.813 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 17 MB/s | 38 MB 00:02 2023-10-30T11:54:24.814 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:54:24.814 INFO:teuthology.orchestra.run.smithi150.stdout:Total 41 MB/s | 95 MB 00:02 2023-10-30T11:54:25.004 INFO:teuthology.orchestra.run.smithi148.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 13 MB/s | 5.3 MB 00:00 2023-10-30T11:54:25.495 INFO:teuthology.orchestra.run.smithi148.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 22 MB/s | 20 MB 00:00 2023-10-30T11:54:25.654 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T11:54:26.013 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T11:54:26.013 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T11:54:26.104 INFO:teuthology.orchestra.run.smithi148.stdout:(3/4): kernel-modules-core-5.14.0-378.el9.x86_6 29 MB/s | 32 MB 00:01 2023-10-30T11:54:26.446 INFO:teuthology.orchestra.run.smithi148.stdout:(4/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 20 MB/s | 38 MB 00:01 2023-10-30T11:54:26.446 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:54:26.446 INFO:teuthology.orchestra.run.smithi148.stdout:Total 49 MB/s | 95 MB 00:01 2023-10-30T11:54:27.274 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T11:54:27.275 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T11:54:27.284 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T11:54:27.659 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T11:54:27.660 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T11:54:28.948 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T11:54:28.949 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T11:54:30.156 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T11:54:30.777 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:54:30.913 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:54:31.551 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T11:54:32.167 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:54:32.287 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:54:33.645 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:54:33.840 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:54:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:54:35.213 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:54:38.491 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:54:38.621 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:54:39.796 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:54:40.019 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:54:43.261 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:54:44.676 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:55:12.425 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:55:12.472 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:55:12.907 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:55:12.908 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:55:12.908 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:55:12.908 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:55:13.154 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:55:13.195 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:55:13.299 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:55:13.300 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T11:55:13.300 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T11:55:13.301 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-30T11:55:13.301 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-30T11:55:13.302 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-30T11:55:13.302 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-30T11:55:13.302 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T11:55:13.303 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T11:55:13.625 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:55:13.626 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:55:13.626 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:55:13.626 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:55:13.832 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:55:13.980 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:55:13.981 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:55:13.981 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-30T11:55:13.982 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-30T11:55:13.982 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-30T11:55:13.982 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-30T11:55:13.982 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2023-10-30T11:55:13.983 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:55:14.027 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:55:14.028 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T11:55:14.028 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T11:55:14.028 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-30T11:55:14.028 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-30T11:55:14.029 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-30T11:55:14.029 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-30T11:55:14.029 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T11:55:14.030 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T11:55:14.101 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:55:14.101 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:55:14.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2023-10-30T11:55:14.142 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2023-10-30T11:55:14.142 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2023-10-30T11:55:14.143 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2023-10-30T11:55:14.143 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 68.el9 2023-10-30T11:55:14.143 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2023-10-30T11:55:14.143 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-30T11:55:14.144 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2023-10-30T11:55:14.144 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8264254 2023-10-30T11:55:14.144 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2023-10-30T11:55:14.144 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-30T11:55:14.144 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-30T11:55:14.145 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-30T11:55:14.145 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-30T11:55:14.145 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2023-10-30T11:55:14.145 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2023-10-30T11:55:14.145 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-30T11:55:14.146 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2023-10-30T11:55:14.146 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2023-10-30T11:55:14.146 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T11:55:14.146 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-30T11:55:14.146 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-30T11:55:14.147 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-30T11:55:14.147 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2023-10-30T11:55:14.147 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T11:55:14.147 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2023-10-30T11:55:14.148 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-30T11:55:14.148 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 5.14.0-378.el9.x86_64 2023-10-30T11:55:14.149 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-30T11:55:14.513 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:55:14.655 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:55:14.656 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:55:14.656 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-30T11:55:14.656 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-30T11:55:14.656 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-30T11:55:14.657 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-30T11:55:14.657 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2023-10-30T11:55:14.657 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:55:14.774 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:55:14.775 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:55:14.775 DEBUG:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2023-10-30T11:55:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2023-10-30T11:55:14.818 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2023-10-30T11:55:14.818 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2023-10-30T11:55:14.819 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.06 2023-10-30T11:55:14.819 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 68.el9 2023-10-30T11:55:14.819 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2023-10-30T11:55:14.819 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-30T11:55:14.819 INFO:teuthology.orchestra.run.smithi148.stdout:Group : Unspecified 2023-10-30T11:55:14.820 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 8264254 2023-10-30T11:55:14.820 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2023-10-30T11:55:14.820 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-30T11:55:14.820 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-30T11:55:14.820 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-30T11:55:14.821 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-30T11:55:14.821 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : builder@centos.org 2023-10-30T11:55:14.821 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : CentOS 2023-10-30T11:55:14.821 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-30T11:55:14.821 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2023-10-30T11:55:14.822 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2023-10-30T11:55:14.822 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T11:55:14.822 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-30T11:55:14.822 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-30T11:55:14.822 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-30T11:55:14.823 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2023-10-30T11:55:14.823 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T11:55:14.825 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2023-10-30T11:55:14.827 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-30T11:55:14.827 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 5.14.0-378.el9.x86_64 2023-10-30T11:55:14.827 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-30T11:55:15.509 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2023-10-30T11:55:16.164 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-30T11:55:16.198 INFO:teuthology.orchestra.run.smithi150.stderr:done 2023-10-30T11:55:16.200 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2023-10-30T11:55:16.216 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.JqfRvu5a3E 2023-10-30T11:55:16.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JqfRvu5a3E 2023-10-30T11:55:16.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.JqfRvu5a3E 2023-10-30T11:55:16.424 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.JqfRvu5a3E is 6KB 2023-10-30T11:55:16.473 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.JqfRvu5a3E 2023-10-30T11:55:16.487 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2023-10-30T11:55:16.551 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-30T11:55:16.552 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-30T11:55:16.552 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-30T11:55:16.552 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-30T11:55:16.553 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-30T11:55:16.707 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2023-10-30T11:55:16.859 INFO:teuthology.orchestra.run.smithi148.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-30T11:55:16.893 INFO:teuthology.orchestra.run.smithi148.stderr:done 2023-10-30T11:55:16.895 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2023-10-30T11:55:16.910 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.qCDizMrdfv 2023-10-30T11:55:16.911 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qCDizMrdfv 2023-10-30T11:55:16.980 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.qCDizMrdfv 2023-10-30T11:55:17.115 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.qCDizMrdfv is 6KB 2023-10-30T11:55:17.163 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.qCDizMrdfv 2023-10-30T11:55:17.178 DEBUG:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2023-10-30T11:55:17.242 INFO:teuthology.orchestra.run.smithi148.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-30T11:55:17.243 INFO:teuthology.orchestra.run.smithi148.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-30T11:55:17.243 INFO:teuthology.orchestra.run.smithi148.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-30T11:55:17.243 INFO:teuthology.orchestra.run.smithi148.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-30T11:55:17.244 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-30T11:55:17.395 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2023-10-30T11:55:46.738 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-30T11:55:46.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-30T11:55:46.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:55:47.398 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-30T11:55:47.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T11:55:47.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:56:05.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-10-30T11:56:05.882 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-10-30T11:56:14.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-30T11:56:14.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:56:14.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T11:56:14.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:56:17.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-10-30T11:56:17.946 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-10-30T11:56:29.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-30T11:56:29.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:56:29.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T11:56:29.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:56:47.610 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-10-30T11:57:02.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-30T11:57:02.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:57:04.445 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-10-30T11:57:04.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-30T11:57:04.900 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-378.el9.x86_64"... 2023-10-30T11:57:04.900 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-10-30T11:57:04.914 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-378.el9.x86_64 2023-10-30T11:57:04.914 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-30T11:57:04.915 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T11:57:04.915 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-30T11:57:05.916 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-30T11:57:05.917 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-30T11:57:06.010 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2023-10-30T11:57:06.041 DEBUG:teuthology.parallel:result is None 2023-10-30T11:57:29.952 DEBUG:teuthology.orchestra.remote:timed out 2023-10-30T11:57:44.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T11:57:44.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:57:45.325 DEBUG:teuthology.orchestra.run.smithi148:> true 2023-10-30T11:57:45.703 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T11:57:45.703 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-10-30T11:57:45.704 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2023-10-30T11:57:45.718 INFO:teuthology.orchestra.run.smithi148.stdout:5.14.0-378.el9.x86_64 2023-10-30T11:57:45.719 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-30T11:57:45.719 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T11:57:45.719 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-30T11:57:46.720 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-30T11:57:46.721 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-30T11:57:46.811 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2023-10-30T11:57:46.850 DEBUG:teuthology.parallel:result is None 2023-10-30T11:57:46.851 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T11:57:46.859 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T11:57:46.859 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T11:57:46.862 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T11:57:46.881 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T11:57:46.888 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T11:57:46.902 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T11:57:46.902 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T11:57:46.920 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T11:57:47.018 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T11:57:47.025 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T11:57:47.026 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T11:57:47.028 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T11:57:47.068 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:57:47.073 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:57:47.082 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:57:47.087 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:57:47.090 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T11:57:47.097 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T11:57:47.097 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T11:57:47.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T11:57:47.155 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T11:57:47.166 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T11:57:47.167 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T11:57:47.170 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T11:57:47.211 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:57:47.269 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:57:47.323 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:57:47.405 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:57:47.447 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T11:57:47.447 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T11:57:47.514 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:57:47.555 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:57:47.619 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:57:47.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:57:47.706 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T11:57:47.707 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T11:57:47.775 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2023-10-30T11:57:47.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2023-10-30T11:57:47.824 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T11:57:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T11:57:48.158 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T11:57:48.165 INFO:teuthology.task.internal:Starting timer... 2023-10-30T11:57:48.166 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T11:57:48.190 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T11:57:48.198 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2023-10-30T11:57:48.289 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2023-10-30T11:57:48.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2023-10-30T11:57:48.337 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2023-10-30T11:57:48.339 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T11:57:48.339 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2023-10-30T11:57:48.364 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2023-10-30T11:57:48.364 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2023-10-30T11:57:48.403 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2023-10-30T11:57:48.403 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi148.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2023-10-30T11:57:48.404 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T11:57:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:57:48.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T11:57:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:57:48.471 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T11:57:48.472 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2023-10-30T11:57:48.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2023-10-30T11:57:48.547 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T11:57:48.557 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-30T11:57:48.563 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-30T11:57:48.687 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T11:57:48.704 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T11:57:48.705 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi148.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T12:03:21.415 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi148.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2023-10-30T12:03:21.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T12:03:21.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:03:21.503 DEBUG:teuthology.orchestra.run.smithi148:> true 2023-10-30T12:03:21.568 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T12:03:21.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-30T12:03:21.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:03:21.651 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-10-30T12:03:21.717 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-30T12:03:21.718 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T12:03:21.729 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T12:03:21.730 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T12:03:21.730 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:03:21.733 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T12:03:21.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:03:21.761 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-30T12:03:21.778 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-30T12:03:21.790 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-30T12:03:21.807 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-30T12:03:21.808 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: ntpd: command not found 2023-10-30T12:03:21.821 INFO:teuthology.orchestra.run.smithi148.stdout:506 Cannot talk to daemon 2023-10-30T12:03:21.837 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2023-10-30T12:03:21.837 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-30T12:03:21.851 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2023-10-30T12:03:21.853 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-30T12:03:21.867 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-30T12:03:21.883 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-30T12:03:21.896 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2023-10-30T12:03:21.900 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T12:03:21.900 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2023-10-30T12:03:21.900 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:03:21.900 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:03:21.900 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:03:21.901 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:03:21.926 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2023-10-30T12:03:21.929 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T12:03:21.930 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-10-30T12:03:21.930 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:03:21.930 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:03:21.930 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:03:21.931 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:03:21.932 INFO:teuthology.run_tasks:Running task install... 2023-10-30T12:03:21.940 DEBUG:teuthology.task.install:project ceph 2023-10-30T12:03:21.940 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d'}} 2023-10-30T12:03:21.940 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d'} 2023-10-30T12:03:21.941 INFO:teuthology.task.install:Using flavor: default 2023-10-30T12:03:21.946 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-10-30T12:03:21.947 INFO:teuthology.task.install:extra packages: [] 2023-10-30T12:03:21.947 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'tag': None, 'wait_for_package': False} 2023-10-30T12:03:21.948 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T12:03:21.950 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'tag': None, 'wait_for_package': False} 2023-10-30T12:03:21.950 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T12:03:22.096 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/ 2023-10-30T12:03:22.096 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6839.gcc36dd85 2023-10-30T12:03:22.101 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/ 2023-10-30T12:03:22.101 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6839.gcc36dd85 2023-10-30T12:03:22.377 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-30T12:03:22.377 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:03:22.377 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-30T12:03:22.406 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-10-30T12:03:22.406 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cc36dd85d142c6fdb77515cad8af108cec98a86d/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-30T12:03:22.439 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-30T12:03:22.440 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:03:22.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-30T12:03:22.469 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-10-30T12:03:22.469 DEBUG:teuthology.orchestra.run.smithi148:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cc36dd85d142c6fdb77515cad8af108cec98a86d/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-30T12:03:22.473 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-30T12:03:22.538 DEBUG:teuthology.orchestra.run.smithi148:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-30T12:03:22.553 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-30T12:03:22.619 DEBUG:teuthology.orchestra.run.smithi148:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-30T12:03:22.619 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2023-10-30T12:03:22.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2023-10-30T12:03:22.684 INFO:teuthology.orchestra.run.smithi148.stdout:check_obsoletes = 1 2023-10-30T12:03:22.686 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum clean all 2023-10-30T12:03:22.860 INFO:teuthology.orchestra.run.smithi150.stdout:72 files removed 2023-10-30T12:03:22.887 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw 2023-10-30T12:03:22.922 INFO:teuthology.orchestra.run.smithi148.stdout:72 files removed 2023-10-30T12:03:22.950 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-radosgw 2023-10-30T12:03:24.072 INFO:teuthology.orchestra.run.smithi148.stdout:ceph packages for x86_64 97 kB/s | 78 kB 00:00 2023-10-30T12:03:24.185 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 80 kB/s | 78 kB 00:00 2023-10-30T12:03:24.948 INFO:teuthology.orchestra.run.smithi148.stdout:ceph noarch packages 11 kB/s | 9.4 kB 00:00 2023-10-30T12:03:25.010 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 12 kB/s | 9.4 kB 00:00 2023-10-30T12:03:25.540 INFO:teuthology.orchestra.run.smithi148.stdout:ceph source packages 3.4 kB/s | 1.9 kB 00:00 2023-10-30T12:03:25.603 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 3.4 kB/s | 1.9 kB 00:00 2023-10-30T12:03:26.042 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-10-30T12:03:26.063 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 15 MB/s | 6.3 MB 00:00 2023-10-30T12:03:27.451 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2023-10-30T12:03:27.464 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2023-10-30T12:03:28.600 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 7.9 MB 00:00 2023-10-30T12:03:29.265 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 7.9 MB 00:01 2023-10-30T12:03:31.562 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 18 MB 00:01 2023-10-30T12:03:36.402 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 5.7 MB 00:01 2023-10-30T12:03:38.153 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2023-10-30T12:03:44.445 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 3.1 MB/s | 19 MB 00:06 2023-10-30T12:03:45.896 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 18 MB 00:15 2023-10-30T12:03:48.995 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2023-10-30T12:03:49.888 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 5.7 MB 00:00 2023-10-30T12:03:51.379 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:03:51.388 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:03:51.388 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-10-30T12:03:51.389 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:03:51.389 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:03:51.390 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 8.7 M 2023-10-30T12:03:51.390 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrading: 2023-10-30T12:03:51.390 INFO:teuthology.orchestra.run.smithi148.stdout: librados2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.3 M 2023-10-30T12:03:51.391 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T12:03:51.391 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-10-30T12:03:51.391 INFO:teuthology.orchestra.run.smithi148.stdout: boost-program-options 2023-10-30T12:03:51.392 INFO:teuthology.orchestra.run.smithi148.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-30T12:03:51.392 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.2 M 2023-10-30T12:03:51.392 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T12:03:51.393 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 24 k 2023-10-30T12:03:51.393 INFO:teuthology.orchestra.run.smithi148.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-30T12:03:51.394 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-30T12:03:51.394 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-30T12:03:51.394 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-30T12:03:51.395 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 715 k 2023-10-30T12:03:51.395 INFO:teuthology.orchestra.run.smithi148.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-30T12:03:51.395 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-30T12:03:51.396 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-30T12:03:51.396 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 472 k 2023-10-30T12:03:51.396 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-30T12:03:51.396 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.8 M 2023-10-30T12:03:51.396 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-30T12:03:51.397 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-30T12:03:51.397 INFO:teuthology.orchestra.run.smithi148.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-30T12:03:51.397 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse 2023-10-30T12:03:51.397 INFO:teuthology.orchestra.run.smithi148.stdout: x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 k 2023-10-30T12:03:51.397 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T12:03:51.397 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 162 k 2023-10-30T12:03:51.398 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-30T12:03:51.398 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 321 k 2023-10-30T12:03:51.398 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 298 k 2023-10-30T12:03:51.398 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 99 k 2023-10-30T12:03:51.399 INFO:teuthology.orchestra.run.smithi148.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-30T12:03:51.399 INFO:teuthology.orchestra.run.smithi148.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-30T12:03:51.399 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:03:51.399 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:03:51.399 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:03:51.399 INFO:teuthology.orchestra.run.smithi148.stdout:Install 28 Packages 2023-10-30T12:03:51.400 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrade 2 Packages 2023-10-30T12:03:51.400 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:03:51.400 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 55 M 2023-10-30T12:03:51.401 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:03:51.673 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2023-10-30T12:03:52.355 INFO:teuthology.orchestra.run.smithi148.stdout:(1/30): ceph-base-18.0.0-6839.gcc36dd85.el9.x86 9.3 MB/s | 5.2 MB 00:00 2023-10-30T12:03:52.421 INFO:teuthology.orchestra.run.smithi148.stdout:(2/30): ceph-selinux-18.0.0-6839.gcc36dd85.el9. 361 kB/s | 24 kB 00:00 2023-10-30T12:03:52.530 INFO:teuthology.orchestra.run.smithi148.stdout:(3/30): libcephfs2-18.0.0-6839.gcc36dd85.el9.x8 6.5 MB/s | 715 kB 00:00 2023-10-30T12:03:52.614 INFO:teuthology.orchestra.run.smithi148.stdout:(4/30): libradosstriper1-18.0.0-6839.gcc36dd85. 5.5 MB/s | 472 kB 00:00 2023-10-30T12:03:53.014 INFO:teuthology.orchestra.run.smithi148.stdout:(5/30): librgw2-18.0.0-6839.gcc36dd85.el9.x86_6 12 MB/s | 4.8 MB 00:00 2023-10-30T12:03:53.081 INFO:teuthology.orchestra.run.smithi148.stdout:(6/30): python3-ceph-argparse-18.0.0-6839.gcc36 681 kB/s | 45 kB 00:00 2023-10-30T12:03:53.148 INFO:teuthology.orchestra.run.smithi148.stdout:(7/30): python3-ceph-common-18.0.0-6839.gcc36dd 1.9 MB/s | 127 kB 00:00 2023-10-30T12:03:53.365 INFO:teuthology.orchestra.run.smithi148.stdout:(8/30): ceph-common-18.0.0-6839.gcc36dd85.el9.x 12 MB/s | 19 MB 00:01 2023-10-30T12:03:53.390 INFO:teuthology.orchestra.run.smithi148.stdout:(9/30): python3-cephfs-18.0.0-6839.gcc36dd85.el 669 kB/s | 162 kB 00:00 2023-10-30T12:03:53.415 INFO:teuthology.orchestra.run.smithi148.stdout:(10/30): python3-rados-18.0.0-6839.gcc36dd85.el 6.3 MB/s | 321 kB 00:00 2023-10-30T12:03:53.441 INFO:teuthology.orchestra.run.smithi148.stdout:(11/30): python3-rbd-18.0.0-6839.gcc36dd85.el9. 5.8 MB/s | 298 kB 00:00 2023-10-30T12:03:53.474 INFO:teuthology.orchestra.run.smithi148.stdout:(12/30): python3-rgw-18.0.0-6839.gcc36dd85.el9. 1.7 MB/s | 99 kB 00:00 2023-10-30T12:03:53.583 INFO:teuthology.orchestra.run.smithi148.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2023-10-30T12:03:53.608 INFO:teuthology.orchestra.run.smithi148.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2023-10-30T12:03:53.642 INFO:teuthology.orchestra.run.smithi148.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-10-30T12:03:53.667 INFO:teuthology.orchestra.run.smithi148.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-10-30T12:03:53.684 INFO:teuthology.orchestra.run.smithi148.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 990 kB/s | 41 kB 00:00 2023-10-30T12:03:53.742 INFO:teuthology.orchestra.run.smithi148.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 601 kB/s | 45 kB 00:00 2023-10-30T12:03:54.218 INFO:teuthology.orchestra.run.smithi148.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 208 kB/s | 111 kB 00:00 2023-10-30T12:03:54.951 INFO:teuthology.orchestra.run.smithi148.stdout:(20/30): libarrow-9.0.0-9.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2023-10-30T12:03:55.085 INFO:teuthology.orchestra.run.smithi148.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 183 kB/s | 24 kB 00:00 2023-10-30T12:03:55.185 INFO:teuthology.orchestra.run.smithi148.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2023-10-30T12:03:55.285 INFO:teuthology.orchestra.run.smithi148.stdout:(23/30): ceph-radosgw-18.0.0-6839.gcc36dd85.el9 2.5 MB/s | 8.7 MB 00:03 2023-10-30T12:03:55.310 INFO:teuthology.orchestra.run.smithi148.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 537 kB/s | 67 kB 00:00 2023-10-30T12:03:55.411 INFO:teuthology.orchestra.run.smithi148.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2023-10-30T12:03:55.436 INFO:teuthology.orchestra.run.smithi148.stdout:(26/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 182 kB/s | 309 kB 00:01 2023-10-30T12:03:55.612 INFO:teuthology.orchestra.run.smithi148.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2023-10-30T12:03:55.737 INFO:teuthology.orchestra.run.smithi148.stdout:(28/30): librados2-18.0.0-6839.gcc36dd85.el9.x8 11 MB/s | 3.3 MB 00:00 2023-10-30T12:03:55.929 INFO:teuthology.orchestra.run.smithi148.stdout:(29/30): librbd1-18.0.0-6839.gcc36dd85.el9.x86_ 9.6 MB/s | 3.0 MB 00:00 2023-10-30T12:03:57.270 INFO:teuthology.orchestra.run.smithi148.stdout:(30/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 422 kB/s | 837 kB 00:01 2023-10-30T12:03:57.278 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:03:57.279 INFO:teuthology.orchestra.run.smithi148.stdout:Total 9.3 MB/s | 55 MB 00:05 2023-10-30T12:03:57.290 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:03:57.328 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:03:57.328 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:03:57.803 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:03:57.803 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:03:59.180 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:03:59.258 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-30T12:03:59.382 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-30T12:03:59.476 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-30T12:03:59.801 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-30T12:03:59.849 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T12:03:59.941 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T12:04:00.025 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/32 2023-10-30T12:04:00.051 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T12:04:00.119 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T12:04:00.142 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-30T12:04:00.176 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 9/32 2023-10-30T12:04:00.243 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T12:04:00.270 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T12:04:00.326 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T12:04:00.432 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-30T12:04:00.467 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-30T12:04:00.529 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-30T12:04:00.558 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-30T12:04:00.586 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-30T12:04:00.612 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-30T12:04:00.645 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T12:04:00.951 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-30T12:04:01.010 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T12:04:01.102 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T12:04:01.135 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 21/32 2023-10-30T12:04:01.177 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-30T12:04:01.603 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-30T12:04:01.631 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T12:04:01.702 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T12:04:01.768 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 25/32 2023-10-30T12:04:01.800 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 26/32 2023-10-30T12:04:03.970 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T12:04:04.005 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T12:04:04.563 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T12:04:04.792 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T12:04:04.910 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T12:04:04.910 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-30T12:04:04.911 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-30T12:04:04.911 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:04.952 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T12:04:06.132 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 19 MB 00:14 2023-10-30T12:04:10.656 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2023-10-30T12:04:13.064 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:04:13.086 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:04:13.086 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-30T12:04:13.086 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:04:13.086 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:04:13.087 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 8.7 M 2023-10-30T12:04:13.087 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2023-10-30T12:04:13.087 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.3 M 2023-10-30T12:04:13.087 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T12:04:13.087 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-30T12:04:13.088 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options 2023-10-30T12:04:13.088 INFO:teuthology.orchestra.run.smithi150.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-30T12:04:13.088 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.2 M 2023-10-30T12:04:13.088 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T12:04:13.088 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 24 k 2023-10-30T12:04:13.089 INFO:teuthology.orchestra.run.smithi150.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-30T12:04:13.089 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-30T12:04:13.089 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-30T12:04:13.089 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-30T12:04:13.089 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 715 k 2023-10-30T12:04:13.089 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-30T12:04:13.090 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-30T12:04:13.090 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-30T12:04:13.090 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 472 k 2023-10-30T12:04:13.090 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-30T12:04:13.091 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.8 M 2023-10-30T12:04:13.091 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-30T12:04:13.091 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-30T12:04:13.091 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-30T12:04:13.091 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse 2023-10-30T12:04:13.092 INFO:teuthology.orchestra.run.smithi150.stdout: x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 k 2023-10-30T12:04:13.092 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T12:04:13.092 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 162 k 2023-10-30T12:04:13.092 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-30T12:04:13.092 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 321 k 2023-10-30T12:04:13.093 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 298 k 2023-10-30T12:04:13.093 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 99 k 2023-10-30T12:04:13.093 INFO:teuthology.orchestra.run.smithi150.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-30T12:04:13.093 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-30T12:04:13.093 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:13.094 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:04:13.094 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:04:13.094 INFO:teuthology.orchestra.run.smithi150.stdout:Install 28 Packages 2023-10-30T12:04:13.094 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 2 Packages 2023-10-30T12:04:13.094 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:13.095 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 55 M 2023-10-30T12:04:13.095 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:04:14.373 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T12:04:14.374 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /sys 2023-10-30T12:04:14.374 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /proc 2023-10-30T12:04:14.374 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /mnt 2023-10-30T12:04:14.374 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /var/tmp 2023-10-30T12:04:14.374 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /home 2023-10-30T12:04:14.375 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /root 2023-10-30T12:04:14.375 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /tmp 2023-10-30T12:04:14.375 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:15.157 INFO:teuthology.orchestra.run.smithi150.stdout:(1/30): ceph-base-18.0.0-6839.gcc36dd85.el9.x86 2.7 MB/s | 5.2 MB 00:01 2023-10-30T12:04:15.200 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T12:04:15.216 INFO:teuthology.orchestra.run.smithi150.stdout:(2/30): ceph-selinux-18.0.0-6839.gcc36dd85.el9. 414 kB/s | 24 kB 00:00 2023-10-30T12:04:15.223 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T12:04:15.223 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:04:15.224 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-30T12:04:15.224 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:04:15.224 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:04:15.224 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:15.225 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T12:04:15.311 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T12:04:15.311 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T12:04:15.433 INFO:teuthology.orchestra.run.smithi150.stdout:(3/30): libcephfs2-18.0.0-6839.gcc36dd85.el9.x8 3.2 MB/s | 715 kB 00:00 2023-10-30T12:04:15.567 INFO:teuthology.orchestra.run.smithi150.stdout:(4/30): libradosstriper1-18.0.0-6839.gcc36dd85. 3.5 MB/s | 472 kB 00:00 2023-10-30T12:04:16.443 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T12:04:16.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/32 2023-10-30T12:04:16.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/32 2023-10-30T12:04:16.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/32 2023-10-30T12:04:16.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/32 2023-10-30T12:04:16.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T12:04:16.445 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 6/32 2023-10-30T12:04:16.445 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T12:04:16.445 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 8/32 2023-10-30T12:04:16.445 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 9/32 2023-10-30T12:04:16.445 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T12:04:16.446 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 11/32 2023-10-30T12:04:16.446 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 12/32 2023-10-30T12:04:16.446 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 13/32 2023-10-30T12:04:16.446 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-30T12:04:16.446 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-30T12:04:16.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-30T12:04:16.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-30T12:04:16.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T12:04:16.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-30T12:04:16.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-30T12:04:16.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-30T12:04:16.448 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-30T12:04:16.448 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-30T12:04:16.448 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-30T12:04:16.448 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-30T12:04:16.448 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-30T12:04:16.449 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-30T12:04:16.449 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-30T12:04:16.450 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T12:04:16.450 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-30T12:04:16.450 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/32 2023-10-30T12:04:16.608 INFO:teuthology.orchestra.run.smithi150.stdout:(5/30): librgw2-18.0.0-6839.gcc36dd85.el9.x86_6 4.6 MB/s | 4.8 MB 00:01 2023-10-30T12:04:16.692 INFO:teuthology.orchestra.run.smithi150.stdout:(6/30): python3-ceph-argparse-18.0.0-6839.gcc36 543 kB/s | 45 kB 00:00 2023-10-30T12:04:16.784 INFO:teuthology.orchestra.run.smithi150.stdout:(7/30): python3-ceph-common-18.0.0-6839.gcc36dd 1.4 MB/s | 127 kB 00:00 2023-10-30T12:04:16.884 INFO:teuthology.orchestra.run.smithi150.stdout:(8/30): python3-cephfs-18.0.0-6839.gcc36dd85.el 1.6 MB/s | 162 kB 00:00 2023-10-30T12:04:16.984 INFO:teuthology.orchestra.run.smithi150.stdout:(9/30): python3-rados-18.0.0-6839.gcc36dd85.el9 3.1 MB/s | 321 kB 00:00 2023-10-30T12:04:17.093 INFO:teuthology.orchestra.run.smithi150.stdout:(10/30): python3-rbd-18.0.0-6839.gcc36dd85.el9. 2.7 MB/s | 298 kB 00:00 2023-10-30T12:04:17.193 INFO:teuthology.orchestra.run.smithi150.stdout:(11/30): python3-rgw-18.0.0-6839.gcc36dd85.el9. 992 kB/s | 99 kB 00:00 2023-10-30T12:04:17.277 INFO:teuthology.orchestra.run.smithi150.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 1.2 MB/s | 106 kB 00:00 2023-10-30T12:04:17.378 INFO:teuthology.orchestra.run.smithi150.stdout:(13/30): ceph-radosgw-18.0.0-6839.gcc36dd85.el9 2.1 MB/s | 8.7 MB 00:04 2023-10-30T12:04:17.403 INFO:teuthology.orchestra.run.smithi150.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2023-10-30T12:04:17.445 INFO:teuthology.orchestra.run.smithi150.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2023-10-30T12:04:17.470 INFO:teuthology.orchestra.run.smithi150.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2023-10-30T12:04:17.495 INFO:teuthology.orchestra.run.smithi150.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 829 kB/s | 41 kB 00:00 2023-10-30T12:04:17.583 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T12:04:17.584 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:17.584 INFO:teuthology.orchestra.run.smithi148.stdout:Upgraded: 2023-10-30T12:04:17.584 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.584 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.584 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:04:17.585 INFO:teuthology.orchestra.run.smithi148.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-30T12:04:17.585 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.585 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.585 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.586 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.586 INFO:teuthology.orchestra.run.smithi148.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-30T12:04:17.586 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-30T12:04:17.586 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-30T12:04:17.586 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-30T12:04:17.587 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.587 INFO:teuthology.orchestra.run.smithi148.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-30T12:04:17.587 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-30T12:04:17.587 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-30T12:04:17.587 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.588 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-30T12:04:17.588 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.588 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-30T12:04:17.588 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-30T12:04:17.588 INFO:teuthology.orchestra.run.smithi148.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-30T12:04:17.589 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.589 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.589 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.589 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-30T12:04:17.589 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.590 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.590 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:17.590 INFO:teuthology.orchestra.run.smithi148.stdout: re2-1:20211101-3.el9.x86_64 2023-10-30T12:04:17.590 INFO:teuthology.orchestra.run.smithi148.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-30T12:04:17.590 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:17.591 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:04:17.746 INFO:teuthology.orchestra.run.smithi150.stdout:(18/30): ceph-common-18.0.0-6839.gcc36dd85.el9. 4.2 MB/s | 19 MB 00:04 2023-10-30T12:04:17.950 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-test 2023-10-30T12:04:18.029 INFO:teuthology.orchestra.run.smithi150.stdout:(19/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 81 kB/s | 45 kB 00:00 2023-10-30T12:04:18.288 INFO:teuthology.orchestra.run.smithi150.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 140 kB/s | 111 kB 00:00 2023-10-30T12:04:18.430 INFO:teuthology.orchestra.run.smithi150.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 172 kB/s | 24 kB 00:00 2023-10-30T12:04:18.496 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:30 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:04:18.530 INFO:teuthology.orchestra.run.smithi150.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 408 kB/s | 309 kB 00:00 2023-10-30T12:04:18.580 INFO:teuthology.orchestra.run.smithi150.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 323 kB/s | 49 kB 00:00 2023-10-30T12:04:18.636 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:04:18.637 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:18.637 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-10-30T12:04:18.637 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:18.637 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:04:18.638 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 41 M 2023-10-30T12:04:18.638 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-10-30T12:04:18.638 INFO:teuthology.orchestra.run.smithi148.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-30T12:04:18.638 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 163 k 2023-10-30T12:04:18.639 INFO:teuthology.orchestra.run.smithi148.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-30T12:04:18.639 INFO:teuthology.orchestra.run.smithi148.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-30T12:04:18.639 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-30T12:04:18.639 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:18.639 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:04:18.640 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:18.640 INFO:teuthology.orchestra.run.smithi148.stdout:Install 6 Packages 2023-10-30T12:04:18.640 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:18.641 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 42 M 2023-10-30T12:04:18.641 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 172 M 2023-10-30T12:04:18.641 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:04:18.685 INFO:teuthology.orchestra.run.smithi150.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 437 kB/s | 67 kB 00:00 2023-10-30T12:04:18.842 INFO:teuthology.orchestra.run.smithi148.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2023-10-30T12:04:18.855 INFO:teuthology.orchestra.run.smithi150.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2023-10-30T12:04:18.884 INFO:teuthology.orchestra.run.smithi148.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-10-30T12:04:18.918 INFO:teuthology.orchestra.run.smithi148.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-30T12:04:19.118 INFO:teuthology.orchestra.run.smithi148.stdout:(4/6): jq-1.6-15.el9.x86_64.rpm 935 kB/s | 187 kB 00:00 2023-10-30T12:04:19.139 INFO:teuthology.orchestra.run.smithi150.stdout:(26/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.5 MB/s | 837 kB 00:00 2023-10-30T12:04:19.143 INFO:teuthology.orchestra.run.smithi148.stdout:(5/6): libcephsqlite-18.0.0-6839.gcc36dd85.el9. 370 kB/s | 163 kB 00:00 2023-10-30T12:04:19.231 INFO:teuthology.orchestra.run.smithi150.stdout:(27/30): libarrow-9.0.0-9.el9.x86_64.rpm 3.7 MB/s | 4.4 MB 00:01 2023-10-30T12:04:19.290 INFO:teuthology.orchestra.run.smithi150.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 3.6 MB/s | 1.6 MB 00:00 2023-10-30T12:04:19.815 INFO:teuthology.orchestra.run.smithi150.stdout:(29/30): librados2-18.0.0-6839.gcc36dd85.el9.x8 4.9 MB/s | 3.3 MB 00:00 2023-10-30T12:04:19.907 INFO:teuthology.orchestra.run.smithi150.stdout:(30/30): librbd1-18.0.0-6839.gcc36dd85.el9.x86_ 4.5 MB/s | 3.0 MB 00:00 2023-10-30T12:04:19.913 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:04:19.913 INFO:teuthology.orchestra.run.smithi150.stdout:Total 8.1 MB/s | 55 MB 00:06 2023-10-30T12:04:19.924 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:04:19.962 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:04:19.962 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:04:20.430 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:04:20.431 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:04:21.863 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:04:21.923 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-30T12:04:21.999 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-30T12:04:22.054 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-30T12:04:22.388 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-30T12:04:22.443 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T12:04:22.527 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T12:04:22.611 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/32 2023-10-30T12:04:22.636 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T12:04:22.699 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T12:04:22.727 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-30T12:04:22.763 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 9/32 2023-10-30T12:04:22.828 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T12:04:22.855 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T12:04:22.904 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T12:04:23.018 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-30T12:04:23.047 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-30T12:04:23.087 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-30T12:04:23.119 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-30T12:04:23.138 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-30T12:04:23.165 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-30T12:04:23.198 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T12:04:23.500 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-30T12:04:23.528 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T12:04:23.619 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T12:04:23.649 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 21/32 2023-10-30T12:04:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-30T12:04:24.110 INFO:teuthology.orchestra.run.smithi148.stdout:(6/6): ceph-test-18.0.0-6839.gcc36dd85.el9.x86_ 7.6 MB/s | 41 MB 00:05 2023-10-30T12:04:24.112 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:04:24.112 INFO:teuthology.orchestra.run.smithi148.stdout:Total 7.6 MB/s | 42 MB 00:05 2023-10-30T12:04:24.113 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-30T12:04:24.125 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:04:24.139 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T12:04:24.144 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:04:24.144 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:04:24.193 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T12:04:24.261 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 25/32 2023-10-30T12:04:24.293 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 26/32 2023-10-30T12:04:24.514 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:04:24.515 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:04:25.060 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:04:25.127 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-30T12:04:25.174 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-30T12:04:25.209 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T12:04:25.251 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-30T12:04:25.279 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T12:04:26.399 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T12:04:26.431 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T12:04:26.980 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T12:04:27.027 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T12:04:27.098 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T12:04:27.098 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-30T12:04:27.099 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-30T12:04:27.099 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:27.136 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T12:04:29.155 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T12:04:29.228 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T12:04:31.292 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T12:04:31.292 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/6 2023-10-30T12:04:31.292 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/6 2023-10-30T12:04:31.293 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T12:04:31.293 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-30T12:04:31.293 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-30T12:04:31.662 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-30T12:04:31.662 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:31.662 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:04:31.663 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:31.663 INFO:teuthology.orchestra.run.smithi148.stdout: jq-1.6-15.el9.x86_64 2023-10-30T12:04:31.663 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:31.663 INFO:teuthology.orchestra.run.smithi148.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-30T12:04:31.663 INFO:teuthology.orchestra.run.smithi148.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-30T12:04:31.664 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-30T12:04:31.664 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:31.664 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:04:31.960 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph 2023-10-30T12:04:32.511 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:44 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:04:32.656 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:04:32.659 INFO:teuthology.orchestra.run.smithi148.stdout:========================================================================================= 2023-10-30T12:04:32.659 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-10-30T12:04:32.659 INFO:teuthology.orchestra.run.smithi148.stdout:========================================================================================= 2023-10-30T12:04:32.659 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:04:32.660 INFO:teuthology.orchestra.run.smithi148.stdout: ceph x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 6.5 k 2023-10-30T12:04:32.660 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-10-30T12:04:32.660 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 2.1 M 2023-10-30T12:04:32.660 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.5 M 2023-10-30T12:04:32.660 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 243 k 2023-10-30T12:04:32.661 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.4 M 2023-10-30T12:04:32.661 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 18 M 2023-10-30T12:04:32.661 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-30T12:04:32.661 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-30T12:04:32.661 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-30T12:04:32.662 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-30T12:04:32.662 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-30T12:04:32.662 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-30T12:04:32.662 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-30T12:04:32.663 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-30T12:04:32.663 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-30T12:04:32.663 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-30T12:04:32.663 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-30T12:04:32.663 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-30T12:04:32.663 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-30T12:04:32.664 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-30T12:04:32.664 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-30T12:04:32.664 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-30T12:04:32.664 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-30T12:04:32.664 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-30T12:04:32.665 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-30T12:04:32.665 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-30T12:04:32.665 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-30T12:04:32.665 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-30T12:04:32.665 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-30T12:04:32.666 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-30T12:04:32.666 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-30T12:04:32.666 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-30T12:04:32.666 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-30T12:04:32.666 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:32.667 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:04:32.667 INFO:teuthology.orchestra.run.smithi148.stdout:========================================================================================= 2023-10-30T12:04:32.667 INFO:teuthology.orchestra.run.smithi148.stdout:Install 33 Packages 2023-10-30T12:04:32.667 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:32.668 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 30 M 2023-10-30T12:04:32.668 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 110 M 2023-10-30T12:04:32.668 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:04:33.370 INFO:teuthology.orchestra.run.smithi148.stdout:(1/33): ceph-18.0.0-6839.gcc36dd85.el9.x86_64.r 30 kB/s | 6.5 kB 00:00 2023-10-30T12:04:34.178 INFO:teuthology.orchestra.run.smithi148.stdout:(2/33): ceph-mgr-18.0.0-6839.gcc36dd85.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2023-10-30T12:04:34.936 INFO:teuthology.orchestra.run.smithi148.stdout:(3/33): ceph-mds-18.0.0-6839.gcc36dd85.el9.x86_ 1.2 MB/s | 2.1 MB 00:01 2023-10-30T12:04:35.228 INFO:teuthology.orchestra.run.smithi148.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6839.gcc36 832 kB/s | 243 kB 00:00 2023-10-30T12:04:35.328 INFO:teuthology.orchestra.run.smithi148.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-10-30T12:04:35.362 INFO:teuthology.orchestra.run.smithi148.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2023-10-30T12:04:35.396 INFO:teuthology.orchestra.run.smithi148.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-10-30T12:04:35.438 INFO:teuthology.orchestra.run.smithi148.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-10-30T12:04:35.463 INFO:teuthology.orchestra.run.smithi148.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 4.0 MB/s | 103 kB 00:00 2023-10-30T12:04:35.538 INFO:teuthology.orchestra.run.smithi148.stdout:(10/33): ceph-mon-18.0.0-6839.gcc36dd85.el9.x86 2.0 MB/s | 4.4 MB 00:02 2023-10-30T12:04:35.563 INFO:teuthology.orchestra.run.smithi148.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 1.2 MB/s | 124 kB 00:00 2023-10-30T12:04:35.589 INFO:teuthology.orchestra.run.smithi148.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-10-30T12:04:35.614 INFO:teuthology.orchestra.run.smithi148.stdout:(13/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-10-30T12:04:35.706 INFO:teuthology.orchestra.run.smithi148.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 558 kB/s | 51 kB 00:00 2023-10-30T12:04:35.823 INFO:teuthology.orchestra.run.smithi148.stdout:(15/33): python3-cryptography-36.0.1-4.el9.x86_ 5.2 MB/s | 1.2 MB 00:00 2023-10-30T12:04:35.882 INFO:teuthology.orchestra.run.smithi148.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 247 kB/s | 43 kB 00:00 2023-10-30T12:04:35.940 INFO:teuthology.orchestra.run.smithi148.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2023-10-30T12:04:35.965 INFO:teuthology.orchestra.run.smithi148.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2023-10-30T12:04:35.991 INFO:teuthology.orchestra.run.smithi148.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2023-10-30T12:04:36.016 INFO:teuthology.orchestra.run.smithi148.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2023-10-30T12:04:36.041 INFO:teuthology.orchestra.run.smithi148.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2023-10-30T12:04:36.066 INFO:teuthology.orchestra.run.smithi148.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2023-10-30T12:04:36.092 INFO:teuthology.orchestra.run.smithi148.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2023-10-30T12:04:36.117 INFO:teuthology.orchestra.run.smithi148.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2023-10-30T12:04:36.142 INFO:teuthology.orchestra.run.smithi148.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2023-10-30T12:04:36.168 INFO:teuthology.orchestra.run.smithi148.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2023-10-30T12:04:36.193 INFO:teuthology.orchestra.run.smithi148.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 330 kB/s | 16 kB 00:00 2023-10-30T12:04:36.218 INFO:teuthology.orchestra.run.smithi148.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2023-10-30T12:04:36.243 INFO:teuthology.orchestra.run.smithi148.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2023-10-30T12:04:36.269 INFO:teuthology.orchestra.run.smithi148.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2023-10-30T12:04:36.294 INFO:teuthology.orchestra.run.smithi148.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.4 MB/s | 427 kB 00:00 2023-10-30T12:04:36.319 INFO:teuthology.orchestra.run.smithi148.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 402 kB/s | 20 kB 00:00 2023-10-30T12:04:36.637 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T12:04:36.637 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2023-10-30T12:04:36.637 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2023-10-30T12:04:36.638 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2023-10-30T12:04:36.638 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2023-10-30T12:04:36.638 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2023-10-30T12:04:36.638 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2023-10-30T12:04:36.638 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2023-10-30T12:04:36.639 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:37.496 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T12:04:37.520 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T12:04:37.520 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:04:37.520 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-30T12:04:37.520 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:04:37.521 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:04:37.521 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:37.522 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T12:04:37.607 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T12:04:37.608 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T12:04:38.687 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T12:04:38.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/32 2023-10-30T12:04:38.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/32 2023-10-30T12:04:38.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/32 2023-10-30T12:04:38.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/32 2023-10-30T12:04:38.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T12:04:38.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 6/32 2023-10-30T12:04:38.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T12:04:38.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 8/32 2023-10-30T12:04:38.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 9/32 2023-10-30T12:04:38.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T12:04:38.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 11/32 2023-10-30T12:04:38.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 12/32 2023-10-30T12:04:38.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 13/32 2023-10-30T12:04:38.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-30T12:04:38.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-30T12:04:38.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-30T12:04:38.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-30T12:04:38.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T12:04:38.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-30T12:04:38.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-30T12:04:38.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-30T12:04:38.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-30T12:04:38.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-30T12:04:38.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-30T12:04:38.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-30T12:04:38.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-30T12:04:38.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-30T12:04:38.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-30T12:04:38.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T12:04:38.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-30T12:04:38.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/32 2023-10-30T12:04:39.061 INFO:teuthology.orchestra.run.smithi148.stdout:(33/33): ceph-osd-18.0.0-6839.gcc36dd85.el9.x86 3.6 MB/s | 18 MB 00:04 2023-10-30T12:04:39.066 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:04:39.067 INFO:teuthology.orchestra.run.smithi148.stdout:Total 4.7 MB/s | 30 MB 00:06 2023-10-30T12:04:39.099 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:04:39.138 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:04:39.139 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:04:39.511 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:04:39.511 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:04:39.662 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T12:04:39.662 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:39.662 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2023-10-30T12:04:39.662 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.663 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.663 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:04:39.663 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-30T12:04:39.663 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.663 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.664 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.664 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.664 INFO:teuthology.orchestra.run.smithi150.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-30T12:04:39.664 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-30T12:04:39.664 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-30T12:04:39.665 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-30T12:04:39.665 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.665 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-30T12:04:39.665 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-30T12:04:39.665 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-30T12:04:39.665 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.666 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-30T12:04:39.666 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.666 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-30T12:04:39.666 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-30T12:04:39.666 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-30T12:04:39.667 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.667 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.667 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.667 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-30T12:04:39.667 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.668 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.668 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:39.668 INFO:teuthology.orchestra.run.smithi150.stdout: re2-1:20211101-3.el9.x86_64 2023-10-30T12:04:39.668 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-30T12:04:39.668 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:39.668 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:04:40.034 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-test 2023-10-30T12:04:40.581 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:30 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:04:40.732 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:04:40.733 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:04:40.733 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-30T12:04:40.733 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:04:40.733 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:04:40.733 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 41 M 2023-10-30T12:04:40.734 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-30T12:04:40.734 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-30T12:04:40.734 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 163 k 2023-10-30T12:04:40.734 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-30T12:04:40.735 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-30T12:04:40.735 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-30T12:04:40.735 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:40.735 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:04:40.735 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:04:40.736 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2023-10-30T12:04:40.736 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:40.736 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 42 M 2023-10-30T12:04:40.737 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 172 M 2023-10-30T12:04:40.737 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:04:40.827 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:04:40.880 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-30T12:04:40.912 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-30T12:04:40.936 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2023-10-30T12:04:40.945 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-30T12:04:40.995 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2023-10-30T12:04:41.029 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-30T12:04:41.037 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-30T12:04:41.115 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-30T12:04:41.156 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-30T12:04:41.162 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): jq-1.6-15.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-10-30T12:04:41.194 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-30T12:04:41.262 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-30T12:04:41.304 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): libcephsqlite-18.0.0-6839.gcc36dd85.el9. 325 kB/s | 163 kB 00:00 2023-10-30T12:04:41.316 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-30T12:04:41.371 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-30T12:04:41.420 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T12:04:41.483 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T12:04:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-30T12:04:41.725 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-30T12:04:41.763 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-30T12:04:41.806 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-30T12:04:41.836 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-30T12:04:41.913 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-30T12:04:41.946 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-30T12:04:41.980 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-30T12:04:42.013 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-30T12:04:42.047 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-30T12:04:42.137 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-30T12:04:42.178 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-30T12:04:42.251 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-30T12:04:42.388 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-30T12:04:42.521 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T12:04:42.685 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 28/33 2023-10-30T12:04:42.716 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T12:04:42.740 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T12:04:42.741 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:04:42.741 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-30T12:04:42.741 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T12:04:42.741 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T12:04:42.742 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:44.250 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T12:04:44.277 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T12:04:44.277 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:04:44.277 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-30T12:04:44.277 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T12:04:44.277 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T12:04:44.278 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:44.718 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T12:04:44.741 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T12:04:44.741 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:04:44.741 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-30T12:04:44.741 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T12:04:44.741 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T12:04:44.742 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:44.943 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T12:04:44.966 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T12:04:44.966 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:04:44.967 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-30T12:04:44.967 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T12:04:44.967 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T12:04:44.967 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:45.033 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T12:04:46.775 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T12:04:46.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/33 2023-10-30T12:04:46.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/33 2023-10-30T12:04:46.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/33 2023-10-30T12:04:46.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/33 2023-10-30T12:04:46.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/33 2023-10-30T12:04:46.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 6/33 2023-10-30T12:04:46.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-30T12:04:46.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-30T12:04:46.777 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-30T12:04:46.777 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-30T12:04:46.777 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T12:04:46.777 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T12:04:46.778 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-30T12:04:46.778 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-30T12:04:46.778 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-30T12:04:46.778 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-30T12:04:46.778 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-30T12:04:46.779 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-30T12:04:46.779 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-30T12:04:46.779 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-30T12:04:46.779 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-30T12:04:46.779 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-30T12:04:46.780 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-30T12:04:46.780 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-30T12:04:46.780 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-30T12:04:46.780 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-30T12:04:46.780 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T12:04:46.781 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-30T12:04:46.781 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-30T12:04:46.782 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-30T12:04:46.782 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-30T12:04:46.782 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-30T12:04:47.857 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-30T12:04:47.858 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:47.858 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:04:47.858 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:47.858 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:47.858 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:47.859 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:04:47.859 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:47.859 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:47.859 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-30T12:04:47.859 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-30T12:04:47.860 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-30T12:04:47.860 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-30T12:04:47.860 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-30T12:04:47.860 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-30T12:04:47.860 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-30T12:04:47.861 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-30T12:04:47.861 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-30T12:04:47.861 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-30T12:04:47.861 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-30T12:04:47.861 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-30T12:04:47.861 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-30T12:04:47.862 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-30T12:04:47.862 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-30T12:04:47.862 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ply-3.11-14.el9.noarch 2023-10-30T12:04:47.862 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-30T12:04:47.862 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-30T12:04:47.863 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-30T12:04:47.863 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-30T12:04:47.863 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-30T12:04:47.863 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-30T12:04:47.863 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-30T12:04:47.864 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-30T12:04:47.864 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-30T12:04:47.864 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-30T12:04:47.864 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-30T12:04:47.864 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:47.864 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:04:48.197 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-base 2023-10-30T12:04:48.755 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:00 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:04:48.846 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:04:48.894 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:04:48.895 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-30T12:04:48.895 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:04:48.958 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install cephadm 2023-10-30T12:04:49.508 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:01 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:04:49.647 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:04:49.648 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-10-30T12:04:49.648 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:49.648 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:04:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 270 k 2023-10-30T12:04:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:49.649 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:04:49.649 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:49.649 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:04:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:49.650 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 270 k 2023-10-30T12:04:49.650 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 265 k 2023-10-30T12:04:49.650 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:04:49.989 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): ceph-test-18.0.0-6839.gcc36dd85.el9.x86_ 4.5 MB/s | 41 MB 00:09 2023-10-30T12:04:49.990 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:04:49.990 INFO:teuthology.orchestra.run.smithi150.stdout:Total 4.5 MB/s | 42 MB 00:09 2023-10-30T12:04:50.004 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:04:50.025 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:04:50.025 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:04:50.300 INFO:teuthology.orchestra.run.smithi148.stdout:cephadm-18.0.0-6839.gcc36dd85.el9.noarch.rpm 414 kB/s | 270 kB 00:00 2023-10-30T12:04:50.301 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:04:50.301 INFO:teuthology.orchestra.run.smithi148.stdout:Total 413 kB/s | 270 kB 00:00 2023-10-30T12:04:50.301 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:04:50.306 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:04:50.307 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:04:50.309 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:04:50.310 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:04:50.381 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:04:50.389 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:04:50.389 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:04:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:04:50.888 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:04:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:04:51.008 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-30T12:04:51.052 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-30T12:04:51.086 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T12:04:51.128 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-30T12:04:51.156 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T12:04:51.240 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:04:51.529 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:04:51.530 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:51.530 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:04:51.530 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:04:51.530 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:51.530 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:04:51.705 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-immutable-object-cache 2023-10-30T12:04:52.265 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:04 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:04:52.409 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:04:52.410 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:52.410 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:04:52.411 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:52.411 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:04:52.411 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 141 k 2023-10-30T12:04:52.411 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:52.411 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:04:52.412 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:52.412 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:04:52.412 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:52.412 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 141 k 2023-10-30T12:04:52.412 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 395 k 2023-10-30T12:04:52.413 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:04:52.764 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-immutable-object-cache-18.0.0-6839.gcc36dd 399 kB/s | 141 kB 00:00 2023-10-30T12:04:52.765 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:04:52.765 INFO:teuthology.orchestra.run.smithi148.stdout:Total 398 kB/s | 141 kB 00:00 2023-10-30T12:04:52.765 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:04:52.772 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:04:52.772 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:04:52.824 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:04:52.825 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:04:52.967 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:04:53.032 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T12:04:53.056 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T12:04:53.056 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:04:53.056 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-30T12:04:53.056 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:54.052 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T12:04:54.053 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:54.053 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:04:54.053 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:54.053 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:54.054 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:04:54.284 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr 2023-10-30T12:04:54.846 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:06 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:04:54.936 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:04:54.967 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T12:04:54.985 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:04:54.986 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-30T12:04:54.986 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:04:55.038 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T12:04:55.048 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-dashboard 2023-10-30T12:04:55.591 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:07 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:04:55.731 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:04:55.732 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:55.732 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-10-30T12:04:55.733 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:55.733 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:04:55.733 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 1.8 M 2023-10-30T12:04:55.733 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-10-30T12:04:55.733 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 23 k 2023-10-30T12:04:55.733 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 15 k 2023-10-30T12:04:55.734 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-30T12:04:55.734 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-30T12:04:55.734 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-30T12:04:55.734 INFO:teuthology.orchestra.run.smithi148.stdout:Installing weak dependencies: 2023-10-30T12:04:55.734 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-30T12:04:55.735 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:55.735 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:04:55.735 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:04:55.735 INFO:teuthology.orchestra.run.smithi148.stdout:Install 7 Packages 2023-10-30T12:04:55.736 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:55.738 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 2.1 M 2023-10-30T12:04:55.739 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 8.6 M 2023-10-30T12:04:55.739 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:04:56.267 INFO:teuthology.orchestra.run.smithi148.stdout:(1/7): ceph-prometheus-alerts-18.0.0-6839.gcc36 42 kB/s | 15 kB 00:00 2023-10-30T12:04:56.293 INFO:teuthology.orchestra.run.smithi148.stdout:(2/7): ceph-grafana-dashboards-18.0.0-6839.gcc3 60 kB/s | 23 kB 00:00 2023-10-30T12:04:56.385 INFO:teuthology.orchestra.run.smithi148.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 77 kB/s | 9.0 kB 00:00 2023-10-30T12:04:56.435 INFO:teuthology.orchestra.run.smithi148.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 288 kB/s | 41 kB 00:00 2023-10-30T12:04:56.494 INFO:teuthology.orchestra.run.smithi148.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 283 kB/s | 31 kB 00:00 2023-10-30T12:04:56.569 INFO:teuthology.orchestra.run.smithi148.stdout:(6/7): ceph-mgr-dashboard-18.0.0-6839.gcc36dd85 2.7 MB/s | 1.8 MB 00:00 2023-10-30T12:04:56.586 INFO:teuthology.orchestra.run.smithi148.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2023-10-30T12:04:56.587 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:04:56.588 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.4 MB/s | 2.1 MB 00:00 2023-10-30T12:04:56.588 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:04:56.596 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:04:56.596 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:04:56.667 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:04:56.667 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:04:56.969 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:04:57.029 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-30T12:04:57.062 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-30T12:04:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T12:04:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/6 2023-10-30T12:04:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/6 2023-10-30T12:04:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T12:04:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-30T12:04:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-30T12:04:57.109 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-30T12:04:57.141 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-30T12:04:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 5/7 2023-10-30T12:04:57.411 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 6/7 2023-10-30T12:04:57.448 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T12:04:57.560 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-30T12:04:57.560 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:57.561 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:04:57.561 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:57.561 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-15.el9.x86_64 2023-10-30T12:04:57.562 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:57.562 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-30T12:04:57.562 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-30T12:04:57.562 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-30T12:04:57.562 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:57.563 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:04:57.846 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph 2023-10-30T12:04:58.019 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T12:04:58.019 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 1/7 2023-10-30T12:04:58.019 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 2/7 2023-10-30T12:04:58.019 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 3/7 2023-10-30T12:04:58.020 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-30T12:04:58.020 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-30T12:04:58.020 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-30T12:04:58.318 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-30T12:04:58.318 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:58.319 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:04:58.319 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:04:58.319 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:04:58.319 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:04:58.320 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-30T12:04:58.320 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-30T12:04:58.320 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-30T12:04:58.320 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-30T12:04:58.320 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:58.320 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:04:58.405 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:48 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:04:58.519 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-30T12:04:58.558 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:04:58.560 INFO:teuthology.orchestra.run.smithi150.stdout:========================================================================================= 2023-10-30T12:04:58.560 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-30T12:04:58.561 INFO:teuthology.orchestra.run.smithi150.stdout:========================================================================================= 2023-10-30T12:04:58.561 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:04:58.561 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 6.5 k 2023-10-30T12:04:58.561 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-30T12:04:58.562 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 2.1 M 2023-10-30T12:04:58.562 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.5 M 2023-10-30T12:04:58.562 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 243 k 2023-10-30T12:04:58.562 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.4 M 2023-10-30T12:04:58.562 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 18 M 2023-10-30T12:04:58.563 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-30T12:04:58.563 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-30T12:04:58.563 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-30T12:04:58.563 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-30T12:04:58.563 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-30T12:04:58.564 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-30T12:04:58.564 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-30T12:04:58.564 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-30T12:04:58.564 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-30T12:04:58.564 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-30T12:04:58.564 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-30T12:04:58.565 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-30T12:04:58.565 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-30T12:04:58.565 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-30T12:04:58.565 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-30T12:04:58.565 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-30T12:04:58.566 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-30T12:04:58.566 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-30T12:04:58.566 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-30T12:04:58.566 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-30T12:04:58.566 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-30T12:04:58.567 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-30T12:04:58.567 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-30T12:04:58.567 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-30T12:04:58.567 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-30T12:04:58.567 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-30T12:04:58.568 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-30T12:04:58.568 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:58.568 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:04:58.568 INFO:teuthology.orchestra.run.smithi150.stdout:========================================================================================= 2023-10-30T12:04:58.569 INFO:teuthology.orchestra.run.smithi150.stdout:Install 33 Packages 2023-10-30T12:04:58.569 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:04:58.569 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 30 M 2023-10-30T12:04:58.570 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 110 M 2023-10-30T12:04:58.570 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:04:59.065 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:11 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:04:59.205 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:04:59.206 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================================ 2023-10-30T12:04:59.206 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-10-30T12:04:59.206 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================================ 2023-10-30T12:04:59.207 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:04:59.207 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 7.4 M 2023-10-30T12:04:59.207 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-10-30T12:04:59.207 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-30T12:04:59.207 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-30T12:04:59.208 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-30T12:04:59.208 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-30T12:04:59.208 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-30T12:04:59.208 INFO:teuthology.orchestra.run.smithi148.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-30T12:04:59.208 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-30T12:04:59.209 INFO:teuthology.orchestra.run.smithi148.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-30T12:04:59.209 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-30T12:04:59.209 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-30T12:04:59.209 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-30T12:04:59.209 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:59.209 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:04:59.210 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================================ 2023-10-30T12:04:59.210 INFO:teuthology.orchestra.run.smithi148.stdout:Install 12 Packages 2023-10-30T12:04:59.210 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:04:59.211 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 37 M 2023-10-30T12:04:59.211 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 209 M 2023-10-30T12:04:59.211 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:04:59.314 INFO:teuthology.orchestra.run.smithi150.stdout:(1/33): ceph-18.0.0-6839.gcc36dd85.el9.x86_64.r 17 kB/s | 6.5 kB 00:00 2023-10-30T12:04:59.326 INFO:teuthology.orchestra.run.smithi148.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 271 kB/s | 32 kB 00:00 2023-10-30T12:04:59.359 INFO:teuthology.orchestra.run.smithi148.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 519 kB/s | 17 kB 00:00 2023-10-30T12:04:59.393 INFO:teuthology.orchestra.run.smithi148.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-30T12:04:59.451 INFO:teuthology.orchestra.run.smithi148.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-10-30T12:04:59.518 INFO:teuthology.orchestra.run.smithi148.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.0 MB/s | 205 kB 00:00 2023-10-30T12:04:59.602 INFO:teuthology.orchestra.run.smithi148.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-10-30T12:04:59.677 INFO:teuthology.orchestra.run.smithi148.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2023-10-30T12:04:59.702 INFO:teuthology.orchestra.run.smithi148.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-10-30T12:04:59.778 INFO:teuthology.orchestra.run.smithi148.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 11 MB/s | 803 kB 00:00 2023-10-30T12:04:59.945 INFO:teuthology.orchestra.run.smithi148.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-10-30T12:04:59.970 INFO:teuthology.orchestra.run.smithi148.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2023-10-30T12:05:00.114 INFO:teuthology.orchestra.run.smithi150.stdout:(2/33): ceph-mgr-18.0.0-6839.gcc36dd85.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2023-10-30T12:05:00.839 INFO:teuthology.orchestra.run.smithi150.stdout:(3/33): ceph-mds-18.0.0-6839.gcc36dd85.el9.x86_ 1.1 MB/s | 2.1 MB 00:01 2023-10-30T12:05:01.048 INFO:teuthology.orchestra.run.smithi150.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6839.gcc36 1.1 MB/s | 243 kB 00:00 2023-10-30T12:05:01.149 INFO:teuthology.orchestra.run.smithi150.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-10-30T12:05:01.182 INFO:teuthology.orchestra.run.smithi150.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 956 kB/s | 32 kB 00:00 2023-10-30T12:05:01.215 INFO:teuthology.orchestra.run.smithi150.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-10-30T12:05:01.258 INFO:teuthology.orchestra.run.smithi150.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-10-30T12:05:01.291 INFO:teuthology.orchestra.run.smithi150.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-10-30T12:05:01.325 INFO:teuthology.orchestra.run.smithi150.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-10-30T12:05:01.380 INFO:teuthology.orchestra.run.smithi150.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 2.0 MB/s | 113 kB 00:00 2023-10-30T12:05:01.418 INFO:teuthology.orchestra.run.smithi150.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.0 MB/s | 187 kB 00:00 2023-10-30T12:05:01.478 INFO:teuthology.orchestra.run.smithi148.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 3.3 MB/s | 7.4 MB 00:02 2023-10-30T12:05:01.481 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:01.482 INFO:teuthology.orchestra.run.smithi148.stdout:Total 16 MB/s | 37 MB 00:02 2023-10-30T12:05:01.482 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:05:01.507 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:05:01.507 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:05:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 4.3 MB/s | 1.2 MB 00:00 2023-10-30T12:05:01.934 INFO:teuthology.orchestra.run.smithi150.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 219 kB/s | 51 kB 00:00 2023-10-30T12:05:01.944 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:05:01.945 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:05:02.043 INFO:teuthology.orchestra.run.smithi150.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 401 kB/s | 43 kB 00:00 2023-10-30T12:05:02.118 INFO:teuthology.orchestra.run.smithi150.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2023-10-30T12:05:02.185 INFO:teuthology.orchestra.run.smithi150.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2023-10-30T12:05:02.227 INFO:teuthology.orchestra.run.smithi150.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2023-10-30T12:05:02.269 INFO:teuthology.orchestra.run.smithi150.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2023-10-30T12:05:02.302 INFO:teuthology.orchestra.run.smithi150.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 699 kB/s | 23 kB 00:00 2023-10-30T12:05:02.336 INFO:teuthology.orchestra.run.smithi150.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 585 kB/s | 19 kB 00:00 2023-10-30T12:05:02.370 INFO:teuthology.orchestra.run.smithi150.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 596 kB/s | 20 kB 00:00 2023-10-30T12:05:02.412 INFO:teuthology.orchestra.run.smithi150.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2023-10-30T12:05:02.462 INFO:teuthology.orchestra.run.smithi150.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2023-10-30T12:05:02.516 INFO:teuthology.orchestra.run.smithi150.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 4.9 MB/s | 272 kB 00:00 2023-10-30T12:05:02.554 INFO:teuthology.orchestra.run.smithi150.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 435 kB/s | 16 kB 00:00 2023-10-30T12:05:02.596 INFO:teuthology.orchestra.run.smithi150.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2023-10-30T12:05:02.630 INFO:teuthology.orchestra.run.smithi150.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.1 MB/s | 36 kB 00:00 2023-10-30T12:05:02.633 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:05:02.680 INFO:teuthology.orchestra.run.smithi150.stdout:(29/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2023-10-30T12:05:02.717 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-30T12:05:02.730 INFO:teuthology.orchestra.run.smithi150.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.4 MB/s | 427 kB 00:00 2023-10-30T12:05:02.749 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-30T12:05:02.772 INFO:teuthology.orchestra.run.smithi150.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 481 kB/s | 20 kB 00:00 2023-10-30T12:05:02.842 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-30T12:05:02.886 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-30T12:05:03.072 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T12:05:03.101 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T12:05:03.178 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-30T12:05:03.861 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-30T12:05:03.973 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T12:05:04.098 INFO:teuthology.orchestra.run.smithi150.stdout:(32/33): ceph-mon-18.0.0-6839.gcc36dd85.el9.x86 938 kB/s | 4.4 MB 00:04 2023-10-30T12:05:05.225 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-30T12:05:05.398 INFO:teuthology.orchestra.run.smithi150.stdout:(33/33): ceph-osd-18.0.0-6839.gcc36dd85.el9.x86 3.4 MB/s | 18 MB 00:05 2023-10-30T12:05:05.402 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:05.402 INFO:teuthology.orchestra.run.smithi150.stdout:Total 4.4 MB/s | 30 MB 00:06 2023-10-30T12:05:05.435 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:05.475 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:05.475 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:05.836 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:05.837 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:06.252 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-30T12:05:06.284 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T12:05:07.194 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:07.259 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-30T12:05:07.299 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-30T12:05:07.349 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-30T12:05:07.442 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-30T12:05:07.528 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-30T12:05:07.589 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-30T12:05:07.650 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-30T12:05:07.724 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-30T12:05:07.778 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-30T12:05:07.816 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-30T12:05:07.865 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T12:05:07.927 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T12:05:08.141 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-30T12:05:08.195 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-30T12:05:08.225 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-30T12:05:08.260 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-30T12:05:08.290 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-30T12:05:08.367 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-30T12:05:08.401 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-30T12:05:08.434 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-30T12:05:08.475 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-30T12:05:08.509 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-30T12:05:08.599 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-30T12:05:08.640 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-30T12:05:08.722 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-30T12:05:08.820 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T12:05:08.820 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 1/12 2023-10-30T12:05:08.820 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-30T12:05:08.821 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-30T12:05:08.821 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-30T12:05:08.821 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T12:05:08.821 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T12:05:08.821 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-30T12:05:08.822 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-30T12:05:08.822 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T12:05:08.822 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-30T12:05:08.822 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-30T12:05:08.859 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-30T12:05:08.984 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T12:05:09.161 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 28/33 2023-10-30T12:05:09.187 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T12:05:09.213 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T12:05:09.213 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:05:09.213 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-30T12:05:09.213 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T12:05:09.213 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T12:05:09.214 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:09.330 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-30T12:05:09.331 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:09.331 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:05:09.331 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:09.331 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-30T12:05:09.332 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-30T12:05:09.332 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-30T12:05:09.332 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-30T12:05:09.332 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-30T12:05:09.332 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-30T12:05:09.333 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-30T12:05:09.333 INFO:teuthology.orchestra.run.smithi148.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-30T12:05:09.333 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-30T12:05:09.333 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-30T12:05:09.333 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-30T12:05:09.334 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:09.334 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:09.722 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-rook 2023-10-30T12:05:10.276 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:22 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:10.418 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:10.421 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================== 2023-10-30T12:05:10.422 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-10-30T12:05:10.422 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================== 2023-10-30T12:05:10.422 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:05:10.422 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 48 k 2023-10-30T12:05:10.423 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-10-30T12:05:10.423 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-30T12:05:10.423 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-30T12:05:10.423 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-30T12:05:10.423 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-30T12:05:10.423 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-30T12:05:10.424 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-30T12:05:10.424 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-30T12:05:10.424 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-30T12:05:10.424 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-30T12:05:10.424 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-30T12:05:10.425 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-30T12:05:10.425 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-30T12:05:10.425 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:10.425 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:05:10.425 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================== 2023-10-30T12:05:10.426 INFO:teuthology.orchestra.run.smithi148.stdout:Install 13 Packages 2023-10-30T12:05:10.426 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:10.426 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 2.0 M 2023-10-30T12:05:10.426 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 26 M 2023-10-30T12:05:10.426 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:05:10.609 INFO:teuthology.orchestra.run.smithi148.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 196 kB/s | 24 kB 00:00 2023-10-30T12:05:10.634 INFO:teuthology.orchestra.run.smithi148.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2023-10-30T12:05:10.676 INFO:teuthology.orchestra.run.smithi148.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-10-30T12:05:10.701 INFO:teuthology.orchestra.run.smithi148.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-10-30T12:05:10.726 INFO:teuthology.orchestra.run.smithi148.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-30T12:05:10.738 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T12:05:10.752 INFO:teuthology.orchestra.run.smithi148.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2023-10-30T12:05:10.763 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T12:05:10.764 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:05:10.764 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-30T12:05:10.764 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T12:05:10.764 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T12:05:10.764 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:10.861 INFO:teuthology.orchestra.run.smithi148.stdout:(7/13): ceph-mgr-rook-18.0.0-6839.gcc36dd85.el9 127 kB/s | 48 kB 00:00 2023-10-30T12:05:10.886 INFO:teuthology.orchestra.run.smithi148.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2023-10-30T12:05:11.044 INFO:teuthology.orchestra.run.smithi148.stdout:(9/13): python3-google-auth-2.23.3-1.el9.noarch 1.1 MB/s | 197 kB 00:00 2023-10-30T12:05:11.169 INFO:teuthology.orchestra.run.smithi148.stdout:(10/13): python3-cachetools-4.2.4-1.el9.noarch. 73 kB/s | 32 kB 00:00 2023-10-30T12:05:11.253 INFO:teuthology.orchestra.run.smithi148.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2023-10-30T12:05:11.258 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T12:05:11.283 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T12:05:11.283 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:05:11.283 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-30T12:05:11.283 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T12:05:11.284 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T12:05:11.284 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:11.353 INFO:teuthology.orchestra.run.smithi148.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 489 kB/s | 90 kB 00:00 2023-10-30T12:05:11.462 INFO:teuthology.orchestra.run.smithi148.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 142 kB/s | 59 kB 00:00 2023-10-30T12:05:11.464 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:11.465 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2023-10-30T12:05:11.465 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:05:11.478 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:05:11.478 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:05:11.481 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T12:05:11.504 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T12:05:11.504 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:05:11.504 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-30T12:05:11.505 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T12:05:11.505 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T12:05:11.505 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:11.570 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T12:05:11.573 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:05:11.575 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:05:12.105 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:05:12.266 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-30T12:05:12.322 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-30T12:05:12.386 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-30T12:05:12.423 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-30T12:05:12.464 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-30T12:05:12.566 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-30T12:05:12.676 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-30T12:05:12.724 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-30T12:05:13.246 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T12:05:13.247 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/33 2023-10-30T12:05:13.247 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/33 2023-10-30T12:05:13.247 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/33 2023-10-30T12:05:13.247 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/33 2023-10-30T12:05:13.248 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/33 2023-10-30T12:05:13.248 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 6/33 2023-10-30T12:05:13.248 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-30T12:05:13.248 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-30T12:05:13.248 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-30T12:05:13.249 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-30T12:05:13.249 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T12:05:13.249 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T12:05:13.249 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-30T12:05:13.249 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-30T12:05:13.249 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-30T12:05:13.250 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-30T12:05:13.250 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-30T12:05:13.250 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-30T12:05:13.250 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-30T12:05:13.250 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-30T12:05:13.251 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-30T12:05:13.251 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-30T12:05:13.251 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-30T12:05:13.251 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-30T12:05:13.251 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-30T12:05:13.252 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-30T12:05:13.252 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T12:05:13.252 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-30T12:05:13.252 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-30T12:05:13.252 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-30T12:05:13.252 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-30T12:05:13.253 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-30T12:05:13.453 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-30T12:05:13.512 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-30T12:05:13.546 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-30T12:05:13.591 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-30T12:05:13.616 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T12:05:14.311 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-30T12:05:14.311 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:14.312 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:14.312 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:14.312 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:14.312 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:14.312 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:14.313 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:14.313 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:14.313 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-30T12:05:14.313 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-30T12:05:14.313 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-30T12:05:14.314 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-30T12:05:14.314 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-30T12:05:14.314 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-30T12:05:14.314 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-30T12:05:14.314 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-30T12:05:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-30T12:05:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-30T12:05:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-30T12:05:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-30T12:05:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-30T12:05:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-30T12:05:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-30T12:05:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2023-10-30T12:05:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-30T12:05:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-30T12:05:14.317 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-30T12:05:14.317 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-30T12:05:14.317 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-30T12:05:14.317 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-30T12:05:14.317 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-30T12:05:14.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-30T12:05:14.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-30T12:05:14.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-30T12:05:14.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-30T12:05:14.318 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:14.319 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:14.463 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T12:05:14.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 1/13 2023-10-30T12:05:14.464 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-30T12:05:14.464 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-30T12:05:14.464 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-30T12:05:14.464 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-30T12:05:14.464 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-30T12:05:14.465 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-30T12:05:14.465 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-30T12:05:14.465 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-30T12:05:14.465 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-30T12:05:14.465 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-30T12:05:14.465 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-30T12:05:14.635 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-base 2023-10-30T12:05:14.827 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-30T12:05:14.827 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:14.827 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:05:14.828 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:14.828 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-30T12:05:14.828 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-30T12:05:14.828 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-30T12:05:14.828 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-30T12:05:14.829 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-30T12:05:14.829 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-30T12:05:14.829 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-30T12:05:14.829 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-30T12:05:14.829 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-30T12:05:14.830 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-30T12:05:14.830 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-30T12:05:14.830 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-30T12:05:14.830 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:14.830 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:15.049 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-cephadm 2023-10-30T12:05:15.188 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:05 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:15.279 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:15.327 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:15.328 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-10-30T12:05:15.328 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:15.388 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install cephadm 2023-10-30T12:05:15.600 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:27 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:15.741 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:15.741 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================== 2023-10-30T12:05:15.742 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-10-30T12:05:15.742 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================== 2023-10-30T12:05:15.742 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:05:15.742 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 132 k 2023-10-30T12:05:15.743 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-10-30T12:05:15.743 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-30T12:05:15.743 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-30T12:05:15.743 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-30T12:05:15.743 INFO:teuthology.orchestra.run.smithi148.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-30T12:05:15.744 INFO:teuthology.orchestra.run.smithi148.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-30T12:05:15.744 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:15.744 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:05:15.744 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================== 2023-10-30T12:05:15.744 INFO:teuthology.orchestra.run.smithi148.stdout:Install 6 Packages 2023-10-30T12:05:15.745 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:15.745 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 6.8 M 2023-10-30T12:05:15.745 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 33 M 2023-10-30T12:05:15.746 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:05:15.936 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:05 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:16.055 INFO:teuthology.orchestra.run.smithi148.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.6 MB/s | 228 kB 00:00 2023-10-30T12:05:16.075 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:16.076 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:16.076 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-30T12:05:16.077 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:16.077 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:05:16.077 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 270 k 2023-10-30T12:05:16.077 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:16.077 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:05:16.078 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:16.078 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:05:16.078 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:16.078 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 270 k 2023-10-30T12:05:16.078 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 265 k 2023-10-30T12:05:16.079 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:05:16.172 INFO:teuthology.orchestra.run.smithi148.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2023-10-30T12:05:16.339 INFO:teuthology.orchestra.run.smithi148.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6839.gcc36dd85.e 315 kB/s | 132 kB 00:00 2023-10-30T12:05:16.655 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm-18.0.0-6839.gcc36dd85.el9.noarch.rpm 467 kB/s | 270 kB 00:00 2023-10-30T12:05:16.656 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:16.656 INFO:teuthology.orchestra.run.smithi150.stdout:Total 466 kB/s | 270 kB 00:00 2023-10-30T12:05:16.656 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:16.661 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:16.661 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:16.664 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:16.665 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:16.706 INFO:teuthology.orchestra.run.smithi148.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 113 kB/s | 58 kB 00:00 2023-10-30T12:05:16.735 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:16.840 INFO:teuthology.orchestra.run.smithi148.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 102 kB/s | 51 kB 00:00 2023-10-30T12:05:17.031 INFO:teuthology.orchestra.run.smithi148.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 562 kB/s | 548 kB 00:00 2023-10-30T12:05:17.035 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:17.036 INFO:teuthology.orchestra.run.smithi148.stdout:Total 5.2 MB/s | 6.8 MB 00:01 2023-10-30T12:05:17.036 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:05:17.045 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:05:17.046 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:05:17.134 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:05:17.135 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:05:17.186 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:05:17.267 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:05:17.429 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:05:17.585 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-30T12:05:17.633 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-30T12:05:17.705 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:05:17.989 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:05:17.989 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:17.989 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:17.989 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:17.990 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:17.990 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:18.148 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-30T12:05:18.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-immutable-object-cache 2023-10-30T12:05:18.235 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-30T12:05:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-30T12:05:18.320 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T12:05:18.729 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:08 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:18.875 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:18.876 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:18.876 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:05:18.876 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:18.876 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:05:18.877 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 141 k 2023-10-30T12:05:18.877 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:18.877 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:05:18.877 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:18.877 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:05:18.877 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:18.878 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 141 k 2023-10-30T12:05:18.878 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 395 k 2023-10-30T12:05:18.878 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:05:19.111 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T12:05:19.112 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/6 2023-10-30T12:05:19.112 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-30T12:05:19.112 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-30T12:05:19.112 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-30T12:05:19.113 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-30T12:05:19.415 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-30T12:05:19.416 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:19.416 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:05:19.416 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:19.416 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-30T12:05:19.416 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-30T12:05:19.417 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-30T12:05:19.417 INFO:teuthology.orchestra.run.smithi148.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-30T12:05:19.417 INFO:teuthology.orchestra.run.smithi148.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-30T12:05:19.417 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:19.417 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:19.419 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-immutable-object-cache-18.0.0-6839.gcc36dd 260 kB/s | 141 kB 00:00 2023-10-30T12:05:19.419 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:19.420 INFO:teuthology.orchestra.run.smithi150.stdout:Total 259 kB/s | 141 kB 00:00 2023-10-30T12:05:19.420 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:19.428 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:19.428 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:19.480 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:19.481 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:19.629 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:19.648 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-fuse 2023-10-30T12:05:19.678 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T12:05:19.703 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T12:05:19.703 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:05:19.703 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-30T12:05:19.703 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:20.202 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:32 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:20.343 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:20.344 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:20.344 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-10-30T12:05:20.345 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:20.345 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:05:20.345 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 855 k 2023-10-30T12:05:20.345 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-10-30T12:05:20.345 INFO:teuthology.orchestra.run.smithi148.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-30T12:05:20.345 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:20.346 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:05:20.346 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:20.346 INFO:teuthology.orchestra.run.smithi148.stdout:Install 2 Packages 2023-10-30T12:05:20.346 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:20.346 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 934 k 2023-10-30T12:05:20.347 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 2.7 M 2023-10-30T12:05:20.347 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:05:20.475 INFO:teuthology.orchestra.run.smithi148.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 608 kB/s | 79 kB 00:00 2023-10-30T12:05:20.696 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T12:05:20.697 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:20.697 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:20.697 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:20.697 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:20.698 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:20.900 INFO:teuthology.orchestra.run.smithi148.stdout:(2/2): ceph-fuse-18.0.0-6839.gcc36dd85.el9.x86_ 1.5 MB/s | 855 kB 00:00 2023-10-30T12:05:20.901 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:20.901 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.6 MB/s | 934 kB 00:00 2023-10-30T12:05:20.902 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:05:20.922 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:05:20.922 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:05:20.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr 2023-10-30T12:05:21.001 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:05:21.001 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:05:21.192 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:05:21.331 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-30T12:05:21.408 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T12:05:21.484 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:11 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:21.574 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:21.622 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:21.623 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-10-30T12:05:21.624 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:21.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-dashboard 2023-10-30T12:05:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T12:05:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T12:05:22.229 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:12 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:22.369 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:22.369 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:22.370 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-30T12:05:22.370 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:22.370 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:05:22.370 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 1.8 M 2023-10-30T12:05:22.371 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-30T12:05:22.371 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 23 k 2023-10-30T12:05:22.371 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 15 k 2023-10-30T12:05:22.371 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-30T12:05:22.372 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-30T12:05:22.372 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-30T12:05:22.372 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2023-10-30T12:05:22.372 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-30T12:05:22.372 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:22.372 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:05:22.373 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:22.373 INFO:teuthology.orchestra.run.smithi150.stdout:Install 7 Packages 2023-10-30T12:05:22.373 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:22.374 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.1 M 2023-10-30T12:05:22.374 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 8.6 M 2023-10-30T12:05:22.374 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:05:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-30T12:05:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:22.507 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:05:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-30T12:05:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:22.507 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:22.736 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-volume 2023-10-30T12:05:22.922 INFO:teuthology.orchestra.run.smithi150.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6839.gcc3 125 kB/s | 23 kB 00:00 2023-10-30T12:05:22.989 INFO:teuthology.orchestra.run.smithi150.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6839.gcc36 60 kB/s | 15 kB 00:00 2023-10-30T12:05:23.056 INFO:teuthology.orchestra.run.smithi150.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 68 kB/s | 9.0 kB 00:00 2023-10-30T12:05:23.115 INFO:teuthology.orchestra.run.smithi150.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 535 kB/s | 31 kB 00:00 2023-10-30T12:05:23.165 INFO:teuthology.orchestra.run.smithi150.stdout:(5/7): ceph-mgr-dashboard-18.0.0-6839.gcc36dd85 4.1 MB/s | 1.8 MB 00:00 2023-10-30T12:05:23.190 INFO:teuthology.orchestra.run.smithi150.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 205 kB/s | 41 kB 00:00 2023-10-30T12:05:23.232 INFO:teuthology.orchestra.run.smithi150.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2023-10-30T12:05:23.235 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:23.235 INFO:teuthology.orchestra.run.smithi150.stdout:Total 2.4 MB/s | 2.1 MB 00:00 2023-10-30T12:05:23.236 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:23.243 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:23.243 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:23.293 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:35 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:23.307 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:23.308 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:23.434 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:23.435 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:23.435 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2023-10-30T12:05:23.436 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:23.436 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:05:23.436 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-volume noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 262 k 2023-10-30T12:05:23.437 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:23.437 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:05:23.437 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:23.437 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:05:23.438 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:23.438 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 262 k 2023-10-30T12:05:23.438 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 1.2 M 2023-10-30T12:05:23.439 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:05:23.605 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:23.658 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-30T12:05:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-30T12:05:23.747 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-30T12:05:23.788 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-30T12:05:23.816 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 5/7 2023-10-30T12:05:24.057 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 6/7 2023-10-30T12:05:24.086 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T12:05:24.179 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-volume-18.0.0-6839.gcc36dd85.el9.noarch.rp 353 kB/s | 262 kB 00:00 2023-10-30T12:05:24.179 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:24.180 INFO:teuthology.orchestra.run.smithi148.stdout:Total 352 kB/s | 262 kB 00:00 2023-10-30T12:05:24.180 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:05:24.188 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:05:24.188 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:05:24.225 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:05:24.225 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:05:24.442 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:05:24.506 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:05:24.521 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:05:24.521 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:05:24.521 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-30T12:05:24.522 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:24.637 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T12:05:24.638 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 1/7 2023-10-30T12:05:24.638 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 2/7 2023-10-30T12:05:24.638 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 3/7 2023-10-30T12:05:24.638 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-30T12:05:24.638 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-30T12:05:24.639 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-30T12:05:24.945 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-30T12:05:24.945 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:24.946 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:24.946 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:24.946 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:24.946 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:24.947 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-30T12:05:24.947 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-30T12:05:24.947 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-30T12:05:24.947 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-30T12:05:24.948 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:24.948 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:25.166 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-30T12:05:25.549 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:05:25.549 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:25.550 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:05:25.550 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:25.550 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:25.550 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:25.755 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:15 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:25.847 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados-devel 2023-10-30T12:05:25.899 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:25.900 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2023-10-30T12:05:25.900 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-30T12:05:25.901 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2023-10-30T12:05:25.901 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:05:25.901 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 7.4 M 2023-10-30T12:05:25.901 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-30T12:05:25.901 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-30T12:05:25.901 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-30T12:05:25.902 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-30T12:05:25.902 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-30T12:05:25.902 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-30T12:05:25.902 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-30T12:05:25.903 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-30T12:05:25.903 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-30T12:05:25.903 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-30T12:05:25.903 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-30T12:05:25.903 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-30T12:05:25.903 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:25.904 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:05:25.904 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2023-10-30T12:05:25.904 INFO:teuthology.orchestra.run.smithi150.stdout:Install 12 Packages 2023-10-30T12:05:25.904 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:25.905 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 37 M 2023-10-30T12:05:25.905 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 209 M 2023-10-30T12:05:25.905 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:05:26.006 INFO:teuthology.orchestra.run.smithi150.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 308 kB/s | 32 kB 00:00 2023-10-30T12:05:26.064 INFO:teuthology.orchestra.run.smithi150.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 506 kB/s | 17 kB 00:00 2023-10-30T12:05:26.098 INFO:teuthology.orchestra.run.smithi150.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-30T12:05:26.165 INFO:teuthology.orchestra.run.smithi150.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-10-30T12:05:26.223 INFO:teuthology.orchestra.run.smithi150.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.6 MB/s | 205 kB 00:00 2023-10-30T12:05:26.291 INFO:teuthology.orchestra.run.smithi150.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2023-10-30T12:05:26.366 INFO:teuthology.orchestra.run.smithi150.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2023-10-30T12:05:26.391 INFO:teuthology.orchestra.run.smithi150.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-10-30T12:05:26.406 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:38 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:26.458 INFO:teuthology.orchestra.run.smithi150.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 12 MB/s | 803 kB 00:00 2023-10-30T12:05:26.548 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:26.549 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:26.550 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:05:26.550 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:26.550 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:05:26.550 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T12:05:26.550 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:26.551 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:05:26.551 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:26.551 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:05:26.551 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:26.551 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 127 k 2023-10-30T12:05:26.552 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 456 k 2023-10-30T12:05:26.552 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:05:26.642 INFO:teuthology.orchestra.run.smithi150.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 57 MB/s | 16 MB 00:00 2023-10-30T12:05:26.675 INFO:teuthology.orchestra.run.smithi150.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 910 kB/s | 195 kB 00:00 2023-10-30T12:05:27.252 INFO:teuthology.orchestra.run.smithi148.stdout:librados-devel-18.0.0-6839.gcc36dd85.el9.x86_64 180 kB/s | 127 kB 00:00 2023-10-30T12:05:27.253 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:27.253 INFO:teuthology.orchestra.run.smithi148.stdout:Total 180 kB/s | 127 kB 00:00 2023-10-30T12:05:27.253 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:05:27.259 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:05:27.259 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:05:27.316 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:05:27.316 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:05:27.465 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:05:27.566 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:27.987 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:28.293 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:28.294 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:28.294 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:05:28.294 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:28.294 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:28.294 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:28.524 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs2 2023-10-30T12:05:29.079 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:41 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:29.173 INFO:teuthology.orchestra.run.smithi148.stdout:Package libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:29.225 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:29.226 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-30T12:05:29.226 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:29.285 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs-devel 2023-10-30T12:05:29.829 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:41 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:29.968 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:29.970 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:05:29.970 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:29.970 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:05:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 31 k 2023-10-30T12:05:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:29.971 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:05:29.971 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:29.971 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:05:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:29.972 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 31 k 2023-10-30T12:05:29.972 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 139 k 2023-10-30T12:05:29.972 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:05:30.138 INFO:teuthology.orchestra.run.smithi148.stdout:libcephfs-devel-18.0.0-6839.gcc36dd85.el9.x86_6 186 kB/s | 31 kB 00:00 2023-10-30T12:05:30.139 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:30.139 INFO:teuthology.orchestra.run.smithi148.stdout:Total 184 kB/s | 31 kB 00:00 2023-10-30T12:05:30.139 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:05:30.142 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:05:30.142 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:05:30.166 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:05:30.167 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:05:30.252 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:05:30.359 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:30.723 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:30.984 INFO:teuthology.orchestra.run.smithi150.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 1.5 MB/s | 7.4 MB 00:05 2023-10-30T12:05:30.986 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:30.987 INFO:teuthology.orchestra.run.smithi150.stdout:Total 7.3 MB/s | 37 MB 00:05 2023-10-30T12:05:30.987 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:31.000 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:31.000 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:31.001 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:05:31.001 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:31.001 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:31.001 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:31.013 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:31.013 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:31.171 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados2 2023-10-30T12:05:31.440 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:31.440 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:31.721 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:43 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:31.813 INFO:teuthology.orchestra.run.smithi148.stdout:Package librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:31.864 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:31.865 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-30T12:05:31.865 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:31.929 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librbd1 2023-10-30T12:05:32.140 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:32.209 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-30T12:05:32.238 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-30T12:05:32.339 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-30T12:05:32.383 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-30T12:05:32.478 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:44 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:32.566 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T12:05:32.568 INFO:teuthology.orchestra.run.smithi148.stdout:Package librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:32.598 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T12:05:32.618 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:32.619 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-30T12:05:32.619 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:32.675 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-30T12:05:32.684 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rados 2023-10-30T12:05:33.232 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:45 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:33.322 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:33.324 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-30T12:05:33.371 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:33.372 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-30T12:05:33.372 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T12:05:33.438 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rgw 2023-10-30T12:05:33.984 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:45 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:34.075 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:34.124 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:34.125 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-30T12:05:34.126 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:34.186 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-cephfs 2023-10-30T12:05:34.668 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-30T12:05:34.728 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:46 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:34.819 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:34.869 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:34.870 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-30T12:05:34.870 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:34.932 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rbd 2023-10-30T12:05:35.476 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:47 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:35.566 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:35.615 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:35.616 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-30T12:05:35.616 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:35.683 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-fuse 2023-10-30T12:05:35.702 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-30T12:05:35.730 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T12:05:36.235 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:48 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:36.375 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:36.377 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:36.377 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-10-30T12:05:36.377 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:36.377 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:05:36.378 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 85 k 2023-10-30T12:05:36.378 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:36.378 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:05:36.378 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:36.379 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:05:36.379 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:36.379 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 85 k 2023-10-30T12:05:36.379 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 227 k 2023-10-30T12:05:36.379 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:05:36.849 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-fuse-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 180 kB/s | 85 kB 00:00 2023-10-30T12:05:36.850 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:36.851 INFO:teuthology.orchestra.run.smithi148.stdout:Total 179 kB/s | 85 kB 00:00 2023-10-30T12:05:36.851 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:05:36.858 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:05:36.858 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:05:36.912 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:05:36.913 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:05:37.055 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:05:37.157 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:37.576 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:37.889 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:37.889 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:37.889 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:05:37.889 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:37.890 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:37.890 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:38.129 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-mirror 2023-10-30T12:05:38.256 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T12:05:38.256 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 1/12 2023-10-30T12:05:38.256 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-30T12:05:38.256 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-30T12:05:38.257 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-30T12:05:38.257 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T12:05:38.257 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T12:05:38.257 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-30T12:05:38.257 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-30T12:05:38.257 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T12:05:38.258 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-30T12:05:38.258 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-30T12:05:38.696 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:50 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:38.713 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-30T12:05:38.713 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:38.713 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:38.714 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:38.714 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-30T12:05:38.714 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-30T12:05:38.715 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-30T12:05:38.715 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-30T12:05:38.715 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-30T12:05:38.715 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-30T12:05:38.715 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-30T12:05:38.715 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-30T12:05:38.716 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-30T12:05:38.716 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-30T12:05:38.716 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-30T12:05:38.716 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:38.716 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:38.839 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:38.840 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:38.840 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:05:38.841 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:38.841 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:05:38.841 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T12:05:38.841 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:38.841 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:05:38.842 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:38.842 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:05:38.842 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:38.842 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 3.0 M 2023-10-30T12:05:38.842 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 12 M 2023-10-30T12:05:38.843 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:05:39.082 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-rook 2023-10-30T12:05:39.638 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:29 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:39.781 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:39.782 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-10-30T12:05:39.782 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-30T12:05:39.782 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-10-30T12:05:39.782 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:05:39.783 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 48 k 2023-10-30T12:05:39.783 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-30T12:05:39.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-30T12:05:39.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-30T12:05:39.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-30T12:05:39.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-30T12:05:39.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-30T12:05:39.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-30T12:05:39.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-30T12:05:39.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-30T12:05:39.785 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-30T12:05:39.785 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-30T12:05:39.785 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-30T12:05:39.785 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-30T12:05:39.785 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:39.786 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:05:39.786 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-10-30T12:05:39.786 INFO:teuthology.orchestra.run.smithi150.stdout:Install 13 Packages 2023-10-30T12:05:39.786 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:39.787 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.0 M 2023-10-30T12:05:39.787 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 26 M 2023-10-30T12:05:39.788 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:05:40.105 INFO:teuthology.orchestra.run.smithi150.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 266 kB/s | 24 kB 00:00 2023-10-30T12:05:40.155 INFO:teuthology.orchestra.run.smithi150.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2023-10-30T12:05:40.198 INFO:teuthology.orchestra.run.smithi150.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2023-10-30T12:05:40.222 INFO:teuthology.orchestra.run.smithi150.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-10-30T12:05:40.248 INFO:teuthology.orchestra.run.smithi150.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2023-10-30T12:05:40.273 INFO:teuthology.orchestra.run.smithi150.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2023-10-30T12:05:40.384 INFO:teuthology.orchestra.run.smithi150.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 136 kB/s | 15 kB 00:00 2023-10-30T12:05:40.423 INFO:teuthology.orchestra.run.smithi150.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 183 kB/s | 32 kB 00:00 2023-10-30T12:05:40.474 INFO:teuthology.orchestra.run.smithi150.stdout:(9/13): python3-google-auth-2.23.3-1.el9.noarch 2.2 MB/s | 197 kB 00:00 2023-10-30T12:05:40.499 INFO:teuthology.orchestra.run.smithi150.stdout:(10/13): ceph-mgr-rook-18.0.0-6839.gcc36dd85.el 98 kB/s | 48 kB 00:00 2023-10-30T12:05:40.533 INFO:teuthology.orchestra.run.smithi150.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 9.4 MB/s | 1.0 MB 00:00 2023-10-30T12:05:40.558 INFO:teuthology.orchestra.run.smithi150.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 719 kB/s | 59 kB 00:00 2023-10-30T12:05:40.583 INFO:teuthology.orchestra.run.smithi150.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2023-10-30T12:05:40.586 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:40.586 INFO:teuthology.orchestra.run.smithi150.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2023-10-30T12:05:40.587 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:40.599 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:40.599 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:40.694 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:40.695 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:41.241 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:41.396 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-30T12:05:41.458 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-30T12:05:41.511 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-mirror-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 1.1 MB/s | 3.0 MB 00:02 2023-10-30T12:05:41.512 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:41.512 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.1 MB/s | 3.0 MB 00:02 2023-10-30T12:05:41.513 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:05:41.513 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-30T12:05:41.523 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:05:41.524 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:05:41.552 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-30T12:05:41.593 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-30T12:05:41.599 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:05:41.599 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:05:41.694 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-30T12:05:41.803 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-30T12:05:41.852 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-30T12:05:42.016 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:05:42.078 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:42.103 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:42.103 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:05:42.103 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-30T12:05:42.104 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T12:05:42.104 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T12:05:42.104 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:42.572 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-30T12:05:42.616 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-30T12:05:42.650 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-30T12:05:42.696 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-30T12:05:42.720 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T12:05:43.215 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:43.215 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:43.216 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:05:43.216 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:43.216 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:43.216 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:43.432 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T12:05:43.433 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 1/13 2023-10-30T12:05:43.433 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-30T12:05:43.433 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-30T12:05:43.433 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-30T12:05:43.434 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-30T12:05:43.434 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-30T12:05:43.434 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-30T12:05:43.434 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-30T12:05:43.434 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-30T12:05:43.435 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-30T12:05:43.435 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-30T12:05:43.435 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-30T12:05:43.447 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-nbd 2023-10-30T12:05:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-30T12:05:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:43.786 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-30T12:05:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-30T12:05:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-30T12:05:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-30T12:05:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-30T12:05:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-30T12:05:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-30T12:05:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-30T12:05:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-30T12:05:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-30T12:05:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-30T12:05:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-30T12:05:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:43.789 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:44.005 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:56 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:44.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-cephadm 2023-10-30T12:05:44.146 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:44.147 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:44.147 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-10-30T12:05:44.147 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:44.147 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:05:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 168 k 2023-10-30T12:05:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:44.148 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:05:44.148 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:44.149 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:05:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:44.149 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 168 k 2023-10-30T12:05:44.149 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 486 k 2023-10-30T12:05:44.149 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:05:44.558 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:34 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:44.698 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:44.698 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-10-30T12:05:44.699 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-30T12:05:44.699 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-10-30T12:05:44.699 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:05:44.700 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 132 k 2023-10-30T12:05:44.700 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-30T12:05:44.700 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-30T12:05:44.700 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-30T12:05:44.700 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-30T12:05:44.701 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-30T12:05:44.701 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-30T12:05:44.701 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:44.701 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:05:44.701 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-10-30T12:05:44.702 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2023-10-30T12:05:44.702 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:44.703 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 6.8 M 2023-10-30T12:05:44.703 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 33 M 2023-10-30T12:05:44.703 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:05:44.942 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-10-30T12:05:44.956 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-nbd-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 208 kB/s | 168 kB 00:00 2023-10-30T12:05:44.956 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:44.957 INFO:teuthology.orchestra.run.smithi148.stdout:Total 208 kB/s | 168 kB 00:00 2023-10-30T12:05:44.957 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:05:44.965 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:05:44.965 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:05:45.020 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:05:45.021 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:05:45.034 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2023-10-30T12:05:45.171 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:05:45.218 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 365 kB/s | 58 kB 00:00 2023-10-30T12:05:45.268 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2023-10-30T12:05:45.278 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:45.293 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 683 kB/s | 51 kB 00:00 2023-10-30T12:05:45.510 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): ceph-mgr-cephadm-18.0.0-6839.gcc36dd85.e 182 kB/s | 132 kB 00:00 2023-10-30T12:05:45.513 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:45.513 INFO:teuthology.orchestra.run.smithi150.stdout:Total 8.3 MB/s | 6.8 MB 00:00 2023-10-30T12:05:45.514 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:45.523 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:45.523 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:45.613 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:45.614 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:45.700 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:45.910 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:45.986 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:45.986 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:45.987 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:05:45.987 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:45.987 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:45.987 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:05:46.067 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-30T12:05:46.120 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-30T12:05:46.191 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-base-debuginfo 2023-10-30T12:05:46.645 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-30T12:05:46.730 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-30T12:05:46.741 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:58 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:05:46.791 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-30T12:05:46.824 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T12:05:46.887 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:05:46.888 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:46.889 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:05:46.889 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:46.889 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:05:46.889 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 74 M 2023-10-30T12:05:46.889 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-10-30T12:05:46.890 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 20 M 2023-10-30T12:05:46.890 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:46.890 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:05:46.890 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:05:46.890 INFO:teuthology.orchestra.run.smithi148.stdout:Install 2 Packages 2023-10-30T12:05:46.891 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:05:46.891 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 94 M 2023-10-30T12:05:46.891 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 511 M 2023-10-30T12:05:46.891 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:05:47.630 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T12:05:47.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/6 2023-10-30T12:05:47.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-30T12:05:47.631 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-30T12:05:47.631 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-30T12:05:47.631 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-30T12:05:47.950 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-30T12:05:47.950 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:47.951 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:47.951 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:47.951 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-30T12:05:47.951 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-30T12:05:47.951 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-30T12:05:47.952 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-30T12:05:47.952 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-30T12:05:47.952 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:47.952 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:48.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-fuse 2023-10-30T12:05:48.724 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:38 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:48.865 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:48.866 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:48.867 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-30T12:05:48.867 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:48.867 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:05:48.867 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 855 k 2023-10-30T12:05:48.868 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-30T12:05:48.868 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-30T12:05:48.868 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:48.868 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:05:48.869 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:48.869 INFO:teuthology.orchestra.run.smithi150.stdout:Install 2 Packages 2023-10-30T12:05:48.869 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:48.869 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 934 k 2023-10-30T12:05:48.870 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 2.7 M 2023-10-30T12:05:48.870 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:05:48.970 INFO:teuthology.orchestra.run.smithi150.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 765 kB/s | 79 kB 00:00 2023-10-30T12:05:49.720 INFO:teuthology.orchestra.run.smithi150.stdout:(2/2): ceph-fuse-18.0.0-6839.gcc36dd85.el9.x86_ 1.0 MB/s | 855 kB 00:00 2023-10-30T12:05:49.722 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:49.722 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.1 MB/s | 934 kB 00:00 2023-10-30T12:05:49.722 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:49.744 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:49.745 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:49.825 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:49.825 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:50.027 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:50.152 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-30T12:05:50.222 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T12:05:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T12:05:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T12:05:51.217 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-30T12:05:51.217 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:51.217 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:51.218 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-30T12:05:51.218 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:51.218 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:51.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-volume 2023-10-30T12:05:52.044 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:42 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:52.187 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:52.188 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:52.188 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-30T12:05:52.189 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:52.189 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:05:52.189 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 262 k 2023-10-30T12:05:52.189 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:52.189 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:05:52.190 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:52.190 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:05:52.190 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:52.190 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 262 k 2023-10-30T12:05:52.190 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 1.2 M 2023-10-30T12:05:52.191 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:05:52.852 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-volume-18.0.0-6839.gcc36dd85.el9.noarch.rp 396 kB/s | 262 kB 00:00 2023-10-30T12:05:52.852 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:52.853 INFO:teuthology.orchestra.run.smithi150.stdout:Total 395 kB/s | 262 kB 00:00 2023-10-30T12:05:52.853 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:52.861 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:52.861 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:52.897 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:52.897 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:53.109 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:53.187 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:05:53.203 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:05:53.203 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:05:53.203 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-30T12:05:53.204 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:53.442 INFO:teuthology.orchestra.run.smithi148.stdout:(1/2): ceph-debuginfo-18.0.0-6839.gcc36dd85.el9 3.1 MB/s | 20 MB 00:06 2023-10-30T12:05:54.300 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T12:05:54.301 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:54.301 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:54.301 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T12:05:54.301 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:54.301 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:54.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados-devel 2023-10-30T12:05:55.149 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:45 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:55.288 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:55.289 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:55.290 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:05:55.290 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:55.290 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:05:55.290 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T12:05:55.291 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:55.291 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:05:55.291 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:55.291 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:05:55.291 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:55.292 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 127 k 2023-10-30T12:05:55.292 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 456 k 2023-10-30T12:05:55.292 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:05:55.523 INFO:teuthology.orchestra.run.smithi150.stdout:librados-devel-18.0.0-6839.gcc36dd85.el9.x86_64 544 kB/s | 127 kB 00:00 2023-10-30T12:05:55.524 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:55.525 INFO:teuthology.orchestra.run.smithi150.stdout:Total 541 kB/s | 127 kB 00:00 2023-10-30T12:05:55.525 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:55.529 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:55.530 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:55.585 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:55.585 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:55.727 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:55.813 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:56.263 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:56.563 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:56.564 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:56.564 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:56.564 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:56.564 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:56.565 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:56.805 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs2 2023-10-30T12:05:57.374 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:47 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:57.469 INFO:teuthology.orchestra.run.smithi150.stdout:Package libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:57.519 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:57.519 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-10-30T12:05:57.520 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:57.582 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs-devel 2023-10-30T12:05:58.133 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:48 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:05:58.274 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:05:58.275 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:58.275 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:05:58.275 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:58.276 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:05:58.276 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 31 k 2023-10-30T12:05:58.276 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:58.276 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:05:58.276 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:05:58.277 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:05:58.277 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:58.277 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 31 k 2023-10-30T12:05:58.277 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 139 k 2023-10-30T12:05:58.277 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:05:58.544 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs-devel-18.0.0-6839.gcc36dd85.el9.x86_6 116 kB/s | 31 kB 00:00 2023-10-30T12:05:58.545 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:58.545 INFO:teuthology.orchestra.run.smithi150.stdout:Total 115 kB/s | 31 kB 00:00 2023-10-30T12:05:58.546 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:05:58.548 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:05:58.549 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:05:58.575 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:05:58.575 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:05:58.669 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:05:58.793 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:59.231 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:59.508 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:05:59.508 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:59.509 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:05:59.509 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:59.509 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:05:59.509 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:05:59.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados2 2023-10-30T12:06:00.227 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:50 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:00.318 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:06:00.368 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:00.368 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-10-30T12:06:00.369 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:06:00.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librbd1 2023-10-30T12:06:00.981 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:50 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:01.073 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:06:01.122 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:01.123 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-10-30T12:06:01.123 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:06:01.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rados 2023-10-30T12:06:01.734 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:51 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:01.825 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:06:01.875 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:01.875 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-10-30T12:06:01.876 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:06:01.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rgw 2023-10-30T12:06:02.501 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:52 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:02.590 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:06:02.639 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:02.640 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-10-30T12:06:02.641 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:06:02.704 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-cephfs 2023-10-30T12:06:02.740 INFO:teuthology.orchestra.run.smithi148.stdout:(2/2): ceph-base-debuginfo-18.0.0-6839.gcc36dd8 4.7 MB/s | 74 MB 00:15 2023-10-30T12:06:02.742 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:06:02.743 INFO:teuthology.orchestra.run.smithi148.stdout:Total 5.9 MB/s | 94 MB 00:15 2023-10-30T12:06:02.743 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:06:02.748 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:06:02.748 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:06:03.249 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:53 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:03.339 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:06:03.366 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:06:03.367 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:06:03.390 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:03.391 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-10-30T12:06:03.391 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:06:03.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rbd 2023-10-30T12:06:04.014 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:54 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:04.105 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:06:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-10-30T12:06:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:06:04.222 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-fuse 2023-10-30T12:06:04.775 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:54 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:04.920 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:04.921 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:04.921 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-10-30T12:06:04.921 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:04.921 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:06:04.921 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 85 k 2023-10-30T12:06:04.922 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:04.922 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:06:04.922 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:04.922 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:06:04.923 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:04.923 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 85 k 2023-10-30T12:06:04.923 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 227 k 2023-10-30T12:06:04.923 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:06:05.347 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-fuse-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 199 kB/s | 85 kB 00:00 2023-10-30T12:06:05.348 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:06:05.348 INFO:teuthology.orchestra.run.smithi150.stdout:Total 199 kB/s | 85 kB 00:00 2023-10-30T12:06:05.349 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:06:05.355 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:06:05.356 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:06:05.412 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:06:05.412 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:06:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:06:05.645 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:06:06.059 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:06:06.262 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:06:06.356 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:06:06.357 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:06.357 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:06:06.357 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:06:06.357 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:06.358 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:06:06.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-mirror 2023-10-30T12:06:07.163 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:57 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:07.309 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:07.310 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:07.310 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:06:07.310 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:07.311 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:06:07.311 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T12:06:07.311 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:07.311 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:06:07.311 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:07.312 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:06:07.312 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:07.312 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 3.0 M 2023-10-30T12:06:07.312 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 12 M 2023-10-30T12:06:07.312 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:06:09.009 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-mirror-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 1.8 MB/s | 3.0 MB 00:01 2023-10-30T12:06:09.010 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:06:09.010 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.8 MB/s | 3.0 MB 00:01 2023-10-30T12:06:09.010 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:06:09.021 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:06:09.021 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:06:09.098 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:06:09.099 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:06:09.512 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:06:09.585 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:06:09.609 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:06:09.610 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:06:09.610 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-30T12:06:09.610 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T12:06:09.610 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T12:06:09.611 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:10.765 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:06:10.765 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:10.765 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:06:10.765 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:06:10.766 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:10.766 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:06:11.030 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-nbd 2023-10-30T12:06:11.596 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:01 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:11.737 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:11.738 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:11.739 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-10-30T12:06:11.739 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:11.739 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:06:11.739 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 168 k 2023-10-30T12:06:11.739 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:11.739 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:06:11.740 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:11.740 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:06:11.740 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:11.740 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 168 k 2023-10-30T12:06:11.740 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 486 k 2023-10-30T12:06:11.741 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:06:12.412 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-nbd-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 250 kB/s | 168 kB 00:00 2023-10-30T12:06:12.413 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:06:12.413 INFO:teuthology.orchestra.run.smithi150.stdout:Total 250 kB/s | 168 kB 00:00 2023-10-30T12:06:12.413 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:06:12.421 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:06:12.421 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:06:12.478 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:06:12.478 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:06:12.620 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:06:12.710 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:06:13.154 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:06:13.439 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:06:13.440 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:13.440 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:06:13.440 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:06:13.440 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:13.440 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:06:13.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-base-debuginfo 2023-10-30T12:06:13.936 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T12:06:14.090 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T12:06:14.240 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:04 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:14.384 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:14.385 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:14.385 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:06:14.385 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:14.385 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:06:14.386 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 74 M 2023-10-30T12:06:14.386 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-30T12:06:14.386 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 20 M 2023-10-30T12:06:14.386 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:14.386 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:06:14.387 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:14.387 INFO:teuthology.orchestra.run.smithi150.stdout:Install 2 Packages 2023-10-30T12:06:14.387 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:14.388 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 94 M 2023-10-30T12:06:14.388 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 511 M 2023-10-30T12:06:14.388 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:06:17.493 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T12:06:17.493 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/2 2023-10-30T12:06:17.768 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T12:06:17.768 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:06:17.768 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:06:17.768 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:06:17.768 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:06:17.769 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:06:17.769 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:06:17.966 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-common-debuginfo 2023-10-30T12:06:18.198 INFO:teuthology.orchestra.run.smithi150.stdout:(1/2): ceph-debuginfo-18.0.0-6839.gcc36dd85.el9 5.3 MB/s | 20 MB 00:03 2023-10-30T12:06:18.511 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:30 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:06:18.653 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:06:18.654 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:06:18.654 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:06:18.654 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:06:18.654 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:06:18.655 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 410 M 2023-10-30T12:06:18.655 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:06:18.655 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:06:18.655 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:06:18.655 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:06:18.656 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:06:18.656 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 410 M 2023-10-30T12:06:18.656 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 1.7 G 2023-10-30T12:06:18.656 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:06:23.081 INFO:teuthology.orchestra.run.smithi150.stdout:(2/2): ceph-base-debuginfo-18.0.0-6839.gcc36dd8 8.5 MB/s | 74 MB 00:08 2023-10-30T12:06:23.083 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:06:23.083 INFO:teuthology.orchestra.run.smithi150.stdout:Total 11 MB/s | 94 MB 00:08 2023-10-30T12:06:23.084 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:06:23.088 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:06:23.088 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:06:23.702 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:06:23.703 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:06:26.595 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:06:34.281 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T12:06:34.356 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T12:06:37.766 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T12:06:37.767 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/2 2023-10-30T12:06:38.035 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T12:06:38.036 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:38.036 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:06:38.036 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:06:38.036 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:06:38.037 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:38.037 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:06:38.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-common-debuginfo 2023-10-30T12:06:38.807 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:28 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:06:38.958 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:06:38.959 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:38.959 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:06:38.959 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:38.959 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:06:38.960 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 410 M 2023-10-30T12:06:38.960 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:38.960 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:06:38.960 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:06:38.960 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:06:38.961 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:06:38.961 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 410 M 2023-10-30T12:06:38.961 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 1.7 G 2023-10-30T12:06:38.962 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:06:43.251 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-common-debuginfo-18.0.0-6839.gcc36dd85.el9 17 MB/s | 410 MB 00:24 2023-10-30T12:06:43.252 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:06:43.253 INFO:teuthology.orchestra.run.smithi148.stdout:Total 17 MB/s | 410 MB 00:24 2023-10-30T12:06:43.253 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:06:43.258 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:06:43.258 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:06:45.727 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:06:45.727 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:07:06.342 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-common-debuginfo-18.0.0-6839.gcc36dd85.el9 15 MB/s | 410 MB 00:27 2023-10-30T12:07:06.343 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:06.343 INFO:teuthology.orchestra.run.smithi150.stdout:Total 15 MB/s | 410 MB 00:27 2023-10-30T12:07:06.344 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:07:06.348 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:07:06.348 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:07:08.797 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:07:08.798 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:07:27.389 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:07:27.470 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:35.205 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:35.472 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:35.472 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:07:35.472 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:07:35.472 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:35.473 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:07:35.473 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:07:35.702 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-10-30T12:07:36.243 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:03:48 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:07:36.383 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:07:36.384 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================== 2023-10-30T12:07:36.384 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:07:36.384 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================== 2023-10-30T12:07:36.384 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:07:36.385 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.4 M 2023-10-30T12:07:36.385 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:07:36.385 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:07:36.385 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================== 2023-10-30T12:07:36.385 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:07:36.386 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:07:36.386 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 1.4 M 2023-10-30T12:07:36.386 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 6.3 M 2023-10-30T12:07:36.386 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:07:37.575 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-68 1.2 MB/s | 1.4 MB 00:01 2023-10-30T12:07:37.575 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:37.576 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.2 MB/s | 1.4 MB 00:01 2023-10-30T12:07:37.576 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:07:37.580 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:07:37.580 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:07:37.623 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:07:37.623 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:07:37.889 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:07:37.988 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:07:38.410 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:07:38.679 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:07:38.680 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:07:38.680 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:07:38.680 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:38.680 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:07:38.681 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:07:38.862 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-radosgw-debuginfo 2023-10-30T12:07:39.413 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:03:51 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:07:39.555 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:07:39.556 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:07:39.556 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:07:39.556 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:07:39.556 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:07:39.557 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 174 M 2023-10-30T12:07:39.557 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:07:39.557 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:07:39.557 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:07:39.557 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:07:39.558 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:07:39.558 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 174 M 2023-10-30T12:07:39.558 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 732 M 2023-10-30T12:07:39.558 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:07:50.081 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:07:50.178 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:58.837 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:59.091 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:59.092 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:07:59.092 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:07:59.092 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:59.092 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:07:59.092 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:07:59.311 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-10-30T12:07:59.872 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:49 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:08:00.023 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:08:00.024 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-10-30T12:08:00.024 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:08:00.024 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-10-30T12:08:00.024 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:08:00.024 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.4 M 2023-10-30T12:08:00.025 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:08:00.025 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:08:00.025 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-10-30T12:08:00.025 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:08:00.026 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:08:00.026 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 1.4 M 2023-10-30T12:08:00.027 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 6.3 M 2023-10-30T12:08:00.027 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:08:01.217 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-68 1.2 MB/s | 1.4 MB 00:01 2023-10-30T12:08:01.217 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:08:01.218 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.2 MB/s | 1.4 MB 00:01 2023-10-30T12:08:01.218 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:08:01.221 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:08:01.221 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:08:01.263 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:08:01.263 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:08:01.516 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:08:01.615 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:08:02.047 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:08:02.307 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:08:02.307 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:08:02.308 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:08:02.308 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:08:02.308 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:08:02.308 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:08:02.471 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw-debuginfo 2023-10-30T12:08:03.018 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:53 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:08:03.160 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:08:03.161 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:08:03.161 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:08:03.162 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:08:03.162 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:08:03.162 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 174 M 2023-10-30T12:08:03.162 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:08:03.162 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:08:03.163 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:08:03.163 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:08:03.163 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:08:03.163 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 174 M 2023-10-30T12:08:03.163 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 732 M 2023-10-30T12:08:03.164 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:08:04.366 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-radosgw-debuginfo-18.0.0-6839.gcc36dd85.el 7.0 MB/s | 174 MB 00:24 2023-10-30T12:08:04.367 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:08:04.368 INFO:teuthology.orchestra.run.smithi148.stdout:Total 7.0 MB/s | 174 MB 00:24 2023-10-30T12:08:04.368 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:08:04.372 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:08:04.372 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:08:05.429 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:08:05.430 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:08:23.468 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:08:23.638 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:08:28.124 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:08:28.378 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:08:28.378 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:08:28.378 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:08:28.379 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:08:28.379 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:08:28.379 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:08:28.572 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-test-debuginfo 2023-10-30T12:08:29.121 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:04:41 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:08:29.263 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:08:29.263 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:08:29.264 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:08:29.264 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:08:29.264 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:08:29.264 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 626 M 2023-10-30T12:08:29.265 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:08:29.265 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:08:29.265 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:08:29.265 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:08:29.265 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:08:29.265 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 626 M 2023-10-30T12:08:29.266 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 2.8 G 2023-10-30T12:08:29.266 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:08:40.550 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-radosgw-debuginfo-18.0.0-6839.gcc36dd85.el 4.7 MB/s | 174 MB 00:37 2023-10-30T12:08:40.551 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:08:40.551 INFO:teuthology.orchestra.run.smithi150.stdout:Total 4.7 MB/s | 174 MB 00:37 2023-10-30T12:08:40.551 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:08:40.554 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:08:40.554 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:08:41.612 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:08:41.612 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:08:59.622 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:08:59.746 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:09:04.192 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:09:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:09:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:09:04.441 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:09:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:09:04.442 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:09:04.442 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:09:04.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-test-debuginfo 2023-10-30T12:09:05.174 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:04:55 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:09:05.318 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:09:05.319 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:09:05.319 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:09:05.320 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:09:05.320 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:09:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 626 M 2023-10-30T12:09:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:09:05.320 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:09:05.321 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:09:05.321 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:09:05.321 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:09:05.321 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 626 M 2023-10-30T12:09:05.321 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 2.8 G 2023-10-30T12:09:05.322 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:09:28.968 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-test-debuginfo-18.0.0-6839.gcc36dd85.el9.x 10 MB/s | 626 MB 00:59 2023-10-30T12:09:28.969 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:09:28.969 INFO:teuthology.orchestra.run.smithi148.stdout:Total 10 MB/s | 626 MB 00:59 2023-10-30T12:09:28.969 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:09:28.974 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:09:28.975 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:09:32.702 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:09:32.703 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:10:34.783 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-test-debuginfo-18.0.0-6839.gcc36dd85.el9.x 7.0 MB/s | 626 MB 01:29 2023-10-30T12:10:34.784 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:10:34.785 INFO:teuthology.orchestra.run.smithi150.stdout:Total 7.0 MB/s | 626 MB 01:29 2023-10-30T12:10:34.785 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:10:34.790 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:10:34.790 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:10:38.402 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:10:38.510 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:10:38.510 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:10:38.685 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:10:46.015 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:10:46.284 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:10:46.285 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:10:46.285 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:10:46.285 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:10:46.285 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:10:46.285 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:10:46.542 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-base-debuginfo 2023-10-30T12:10:47.087 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:06:59 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:10:47.177 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:10:47.228 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:10:47.229 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-30T12:10:47.229 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:10:47.285 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-debuginfo 2023-10-30T12:10:47.826 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:06:59 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:10:47.967 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:10:47.968 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:10:47.968 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:10:47.968 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:10:47.969 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:10:47.969 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T12:10:47.969 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:10:47.969 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:10:47.969 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:10:47.970 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:10:47.970 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:10:47.970 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 19 M 2023-10-30T12:10:47.970 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 101 M 2023-10-30T12:10:47.970 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:10:50.329 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-mgr-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 8.1 MB/s | 19 MB 00:02 2023-10-30T12:10:50.330 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:10:50.331 INFO:teuthology.orchestra.run.smithi148.stdout:Total 8.1 MB/s | 19 MB 00:02 2023-10-30T12:10:50.331 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:10:50.335 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:10:50.336 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:10:50.483 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:10:50.483 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:10:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:10:52.820 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:10:53.799 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:10:54.075 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:10:54.075 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:10:54.075 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:10:54.075 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:10:54.076 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:10:54.076 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:10:54.246 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mds-debuginfo 2023-10-30T12:10:54.792 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:07:06 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:10:54.933 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:10:54.934 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:10:54.935 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:10:54.935 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:10:54.935 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:10:54.936 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 47 M 2023-10-30T12:10:54.936 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:10:54.936 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:10:54.937 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:10:54.937 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:10:54.937 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:10:54.937 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 47 M 2023-10-30T12:10:54.938 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 170 M 2023-10-30T12:10:54.938 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:10:59.615 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-mds-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 10 MB/s | 47 MB 00:04 2023-10-30T12:10:59.615 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:10:59.616 INFO:teuthology.orchestra.run.smithi148.stdout:Total 10 MB/s | 47 MB 00:04 2023-10-30T12:10:59.617 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:10:59.620 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:10:59.621 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:10:59.932 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:10:59.933 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:11:04.606 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:11:04.803 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:11:06.174 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:11:06.437 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:11:06.437 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:11:06.437 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:11:06.437 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:11:06.437 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:11:06.438 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:11:06.618 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mon-debuginfo 2023-10-30T12:11:07.164 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:07:19 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:11:07.304 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:11:07.305 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:11:07.305 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:11:07.305 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:11:07.306 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:11:07.306 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 112 M 2023-10-30T12:11:07.306 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:11:07.307 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:11:07.307 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:11:07.307 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:11:07.307 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:11:07.308 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 112 M 2023-10-30T12:11:07.308 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 509 M 2023-10-30T12:11:07.308 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:11:12.425 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-mon-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 22 MB/s | 112 MB 00:05 2023-10-30T12:11:12.426 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:11:12.427 INFO:teuthology.orchestra.run.smithi148.stdout:Total 22 MB/s | 112 MB 00:05 2023-10-30T12:11:12.427 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:11:12.430 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:11:12.431 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:11:13.122 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:11:13.122 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:11:24.860 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:11:24.964 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:11:28.181 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:11:28.465 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:11:28.466 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:11:28.466 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:11:28.466 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:11:28.466 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:11:28.467 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:11:28.628 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-osd-debuginfo 2023-10-30T12:11:29.175 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:07:41 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:11:29.315 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:11:29.315 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:11:29.316 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:11:29.316 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:11:29.316 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:11:29.316 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 315 M 2023-10-30T12:11:29.316 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:11:29.317 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:11:29.317 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:11:29.317 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:11:29.317 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:11:29.317 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 315 M 2023-10-30T12:11:29.318 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 1.4 G 2023-10-30T12:11:29.318 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:11:44.346 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:11:44.893 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:11:52.155 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:11:52.421 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:11:52.422 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:11:52.422 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:11:52.423 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:11:52.423 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:11:52.423 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:11:52.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-base-debuginfo 2023-10-30T12:11:53.229 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:07:43 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:11:53.320 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:11:53.370 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:11:53.371 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-10-30T12:11:53.371 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:11:53.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-debuginfo 2023-10-30T12:11:53.956 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-osd-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 13 MB/s | 315 MB 00:24 2023-10-30T12:11:53.957 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:11:53.958 INFO:teuthology.orchestra.run.smithi148.stdout:Total 13 MB/s | 315 MB 00:24 2023-10-30T12:11:53.958 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:11:53.962 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:11:53.962 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:11:53.985 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:07:43 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:11:54.126 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:11:54.129 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:11:54.130 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:11:54.130 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:11:54.130 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:11:54.131 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T12:11:54.131 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:11:54.131 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:11:54.131 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:11:54.132 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:11:54.132 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:11:54.132 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 19 M 2023-10-30T12:11:54.132 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 101 M 2023-10-30T12:11:54.132 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:11:55.856 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:11:55.857 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:11:58.821 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-mgr-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 4.1 MB/s | 19 MB 00:04 2023-10-30T12:11:58.822 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:11:58.822 INFO:teuthology.orchestra.run.smithi150.stdout:Total 4.1 MB/s | 19 MB 00:04 2023-10-30T12:11:58.823 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:11:58.827 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:11:58.827 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:11:58.974 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:11:58.974 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:12:01.164 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:12:01.261 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:02.246 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:02.516 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:02.517 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:02.517 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:12:02.517 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:12:02.517 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:02.519 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:12:02.693 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mds-debuginfo 2023-10-30T12:12:03.237 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:07:53 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:12:03.379 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:12:03.380 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:12:03.380 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:12:03.380 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:12:03.380 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:12:03.380 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 47 M 2023-10-30T12:12:03.381 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:03.381 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:12:03.381 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:12:03.381 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:12:03.381 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:03.382 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 47 M 2023-10-30T12:12:03.382 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 170 M 2023-10-30T12:12:03.382 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:12:09.574 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-mds-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 7.6 MB/s | 47 MB 00:06 2023-10-30T12:12:09.575 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:12:09.575 INFO:teuthology.orchestra.run.smithi150.stdout:Total 7.6 MB/s | 47 MB 00:06 2023-10-30T12:12:09.575 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:12:09.578 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:12:09.579 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:12:09.892 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:12:09.893 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:12:14.598 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:12:14.706 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:16.070 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:16.333 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:16.333 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:16.333 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:12:16.333 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:12:16.334 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:16.334 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:12:16.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mon-debuginfo 2023-10-30T12:12:17.056 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:08:07 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:12:17.197 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:12:17.198 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:12:17.199 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:12:17.199 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:12:17.199 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:12:17.199 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 112 M 2023-10-30T12:12:17.199 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:17.200 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:12:17.200 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:12:17.200 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:12:17.200 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:17.200 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 112 M 2023-10-30T12:12:17.201 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 509 M 2023-10-30T12:12:17.201 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:12:24.258 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-mon-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 16 MB/s | 112 MB 00:07 2023-10-30T12:12:24.259 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:12:24.259 INFO:teuthology.orchestra.run.smithi150.stdout:Total 16 MB/s | 112 MB 00:07 2023-10-30T12:12:24.260 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:12:24.263 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:12:24.264 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:12:24.954 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:12:24.955 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:12:28.288 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:12:28.458 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:36.600 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:36.689 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:12:36.790 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:36.874 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:36.874 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:36.874 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:12:36.875 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:12:36.875 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:36.875 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:12:37.048 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-fuse-debuginfo 2023-10-30T12:12:37.595 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:08:49 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:12:37.736 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:12:37.737 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:37.738 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:12:37.738 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:37.738 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:12:37.738 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.8 M 2023-10-30T12:12:37.739 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:37.739 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:12:37.739 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:37.739 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:12:37.739 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:37.739 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 5.8 M 2023-10-30T12:12:37.740 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 28 M 2023-10-30T12:12:37.740 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:12:39.981 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:40.255 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:12:40.255 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:40.256 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:12:40.256 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:12:40.256 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:40.256 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:12:40.388 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x 2.2 MB/s | 5.8 MB 00:02 2023-10-30T12:12:40.389 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:12:40.390 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.2 MB/s | 5.8 MB 00:02 2023-10-30T12:12:40.390 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:12:40.394 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:12:40.394 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:12:40.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-osd-debuginfo 2023-10-30T12:12:40.465 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:12:40.466 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:12:40.955 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:08:30 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:12:41.094 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:12:41.095 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:12:41.095 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:12:41.095 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:12:41.096 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:12:41.096 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 315 M 2023-10-30T12:12:41.096 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:41.096 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:12:41.096 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:12:41.097 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:12:41.097 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:12:41.097 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 315 M 2023-10-30T12:12:41.097 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 1.4 G 2023-10-30T12:12:41.097 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:12:41.186 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:12:41.295 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:12:41.855 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:12:42.130 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:12:42.130 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:42.130 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:12:42.131 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:12:42.131 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:42.131 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:12:42.290 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados-devel-debuginfo 2023-10-30T12:12:42.841 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:08:54 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:12:42.980 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:12:42.981 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:42.982 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:12:42.982 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:42.982 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:12:42.982 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 108 k 2023-10-30T12:12:42.983 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:42.983 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:12:42.983 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:42.983 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:12:42.983 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:42.983 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 108 k 2023-10-30T12:12:42.984 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 440 k 2023-10-30T12:12:42.984 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:12:43.466 INFO:teuthology.orchestra.run.smithi148.stdout:librados-devel-debuginfo-18.0.0-6839.gcc36dd85. 224 kB/s | 108 kB 00:00 2023-10-30T12:12:43.467 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:12:43.467 INFO:teuthology.orchestra.run.smithi148.stdout:Total 224 kB/s | 108 kB 00:00 2023-10-30T12:12:43.468 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:12:43.470 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:12:43.470 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:12:43.505 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:12:43.505 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:12:43.621 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:12:43.730 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:12:44.112 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:12:44.384 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:12:44.385 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:44.385 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:12:44.385 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:12:44.385 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:44.386 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:12:44.548 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs2-debuginfo 2023-10-30T12:12:45.094 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:08:57 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:12:45.233 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:12:45.234 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:45.235 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:12:45.235 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:45.235 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:12:45.236 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.1 M 2023-10-30T12:12:45.236 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:45.236 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:12:45.237 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:45.237 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:12:45.237 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:45.238 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 5.1 M 2023-10-30T12:12:45.238 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 24 M 2023-10-30T12:12:45.239 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:12:46.652 INFO:teuthology.orchestra.run.smithi148.stdout:libcephfs2-debuginfo-18.0.0-6839.gcc36dd85.el9. 3.6 MB/s | 5.1 MB 00:01 2023-10-30T12:12:46.653 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:12:46.653 INFO:teuthology.orchestra.run.smithi148.stdout:Total 3.6 MB/s | 5.1 MB 00:01 2023-10-30T12:12:46.654 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:12:46.657 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:12:46.657 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:12:46.721 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:12:46.721 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:12:47.373 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:12:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:12:48.011 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:12:48.261 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:12:48.262 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:48.262 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:12:48.262 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:12:48.263 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:48.263 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:12:48.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados2-debuginfo 2023-10-30T12:12:48.984 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:09:00 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:12:49.123 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:12:49.123 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:49.123 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:12:49.123 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:49.124 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:12:49.124 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 M 2023-10-30T12:12:49.124 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:49.124 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:12:49.124 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:12:49.125 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:12:49.125 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:12:49.125 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 45 M 2023-10-30T12:12:49.126 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 255 M 2023-10-30T12:12:49.126 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:12:54.955 INFO:teuthology.orchestra.run.smithi148.stdout:librados2-debuginfo-18.0.0-6839.gcc36dd85.el9.x 7.7 MB/s | 45 MB 00:05 2023-10-30T12:12:54.956 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:12:54.956 INFO:teuthology.orchestra.run.smithi148.stdout:Total 7.7 MB/s | 45 MB 00:05 2023-10-30T12:12:54.956 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:12:54.960 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:12:54.960 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:12:55.270 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:12:55.270 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:13:00.389 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:13:00.493 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:13:02.350 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:13:02.619 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:13:02.619 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:02.619 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:13:02.619 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:02.620 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:02.620 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:13:02.633 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-osd-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 15 MB/s | 315 MB 00:21 2023-10-30T12:13:02.634 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:02.635 INFO:teuthology.orchestra.run.smithi150.stdout:Total 15 MB/s | 315 MB 00:21 2023-10-30T12:13:02.635 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:13:02.639 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:13:02.639 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:13:02.808 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librbd1-debuginfo 2023-10-30T12:13:03.363 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:09:15 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:13:03.510 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:13:03.511 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:03.511 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:13:03.511 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:03.511 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:13:03.512 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T12:13:03.512 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:03.512 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:13:03.512 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:03.512 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:13:03.513 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:03.513 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 51 M 2023-10-30T12:13:03.513 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 298 M 2023-10-30T12:13:03.513 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:13:04.523 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:13:04.524 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:13:15.148 INFO:teuthology.orchestra.run.smithi148.stdout:librbd1-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 4.4 MB/s | 51 MB 00:11 2023-10-30T12:13:15.148 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:15.149 INFO:teuthology.orchestra.run.smithi148.stdout:Total 4.4 MB/s | 51 MB 00:11 2023-10-30T12:13:15.149 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:13:15.154 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:13:15.154 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:13:15.497 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:13:15.498 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:13:21.367 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:13:21.612 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:13:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:13:23.998 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:13:23.999 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:23.999 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:13:23.999 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:23.999 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:24.000 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:13:24.176 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-cephfs-debuginfo 2023-10-30T12:13:24.726 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:09:36 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:13:24.865 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:13:24.866 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:24.867 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:13:24.867 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:24.867 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:13:24.868 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 615 k 2023-10-30T12:13:24.868 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:24.868 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:13:24.868 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:24.869 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:13:24.869 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:24.869 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 615 k 2023-10-30T12:13:24.870 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 2.1 M 2023-10-30T12:13:24.870 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:13:25.266 INFO:teuthology.orchestra.run.smithi148.stdout:python3-cephfs-debuginfo-18.0.0-6839.gcc36dd85. 1.5 MB/s | 615 kB 00:00 2023-10-30T12:13:25.267 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:25.267 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.5 MB/s | 615 kB 00:00 2023-10-30T12:13:25.268 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:13:25.270 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:13:25.271 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:13:25.308 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:13:25.309 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:13:25.473 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:13:25.582 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:13:26.017 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:13:26.292 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:13:26.292 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:26.292 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:13:26.293 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:26.293 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:26.293 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:13:26.483 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rados-debuginfo 2023-10-30T12:13:27.024 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:09:39 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:13:27.162 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:13:27.163 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:27.163 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:13:27.164 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:27.164 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:13:27.164 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 770 k 2023-10-30T12:13:27.164 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:27.165 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:13:27.165 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:27.165 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:13:27.165 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:27.166 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 770 k 2023-10-30T12:13:27.166 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 2.7 M 2023-10-30T12:13:27.166 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:13:28.202 INFO:teuthology.orchestra.run.smithi148.stdout:python3-rados-debuginfo-18.0.0-6839.gcc36dd85.e 741 kB/s | 770 kB 00:01 2023-10-30T12:13:28.203 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:28.204 INFO:teuthology.orchestra.run.smithi148.stdout:Total 740 kB/s | 770 kB 00:01 2023-10-30T12:13:28.204 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:13:28.207 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:13:28.207 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:13:28.246 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:13:28.247 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:13:28.429 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:13:28.525 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:13:28.914 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:13:29.174 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:13:29.174 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:29.174 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:13:29.175 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:29.175 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:29.175 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:13:29.349 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rbd-debuginfo 2023-10-30T12:13:29.892 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:09:41 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:13:30.030 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:13:30.031 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:13:30.032 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:30.032 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:13:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.1 M 2023-10-30T12:13:30.032 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:30.033 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:13:30.033 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:30.033 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:13:30.033 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:30.033 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 1.1 M 2023-10-30T12:13:30.034 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 4.3 M 2023-10-30T12:13:30.034 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:13:30.964 INFO:teuthology.orchestra.run.smithi148.stdout:python3-rbd-debuginfo-18.0.0-6839.gcc36dd85.el9 1.2 MB/s | 1.1 MB 00:00 2023-10-30T12:13:30.965 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:30.965 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.2 MB/s | 1.1 MB 00:00 2023-10-30T12:13:30.965 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:13:30.969 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:13:30.969 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:13:31.010 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:13:31.010 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:13:31.224 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:13:31.338 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:13:31.727 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:13:31.982 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:13:31.983 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:31.983 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:13:31.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:31.983 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:31.983 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:13:32.193 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rgw-debuginfo 2023-10-30T12:13:32.739 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:09:44 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:13:32.878 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:13:32.879 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:32.879 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:13:32.879 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:32.880 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:13:32.880 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 337 k 2023-10-30T12:13:32.880 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:32.880 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:13:32.880 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:32.881 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:13:32.881 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:32.881 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 337 k 2023-10-30T12:13:32.881 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 1.0 M 2023-10-30T12:13:32.881 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:13:33.417 INFO:teuthology.orchestra.run.smithi148.stdout:python3-rgw-debuginfo-18.0.0-6839.gcc36dd85.el9 627 kB/s | 337 kB 00:00 2023-10-30T12:13:33.419 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:33.419 INFO:teuthology.orchestra.run.smithi148.stdout:Total 626 kB/s | 337 kB 00:00 2023-10-30T12:13:33.419 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:13:33.422 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:13:33.422 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:13:33.458 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:13:33.459 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:13:33.605 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:13:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:13:34.118 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:13:34.382 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:13:34.383 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:34.383 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:13:34.383 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:34.383 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:34.384 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:13:34.560 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-fuse-debuginfo 2023-10-30T12:13:35.106 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:09:47 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:13:35.246 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:13:35.247 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:35.247 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:13:35.247 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:35.247 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:13:35.248 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 506 k 2023-10-30T12:13:35.248 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:35.248 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:13:35.248 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:35.248 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:13:35.248 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:35.249 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 506 k 2023-10-30T12:13:35.249 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 2.5 M 2023-10-30T12:13:35.249 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:13:35.854 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 834 kB/s | 506 kB 00:00 2023-10-30T12:13:35.855 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:35.856 INFO:teuthology.orchestra.run.smithi148.stdout:Total 832 kB/s | 506 kB 00:00 2023-10-30T12:13:35.856 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:13:35.861 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:13:35.861 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:13:35.895 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:13:35.896 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:13:36.058 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:13:36.185 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:13:36.599 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:13:36.739 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:13:36.853 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:13:36.864 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:13:36.865 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:36.865 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:13:36.865 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:36.865 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:36.865 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:13:37.041 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-mirror-debuginfo 2023-10-30T12:13:37.584 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:09:49 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:13:37.723 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:13:37.724 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:37.724 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:13:37.724 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:37.725 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:13:37.725 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T12:13:37.725 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:37.725 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:13:37.725 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:37.726 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:13:37.726 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:37.726 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 51 M 2023-10-30T12:13:37.726 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 289 M 2023-10-30T12:13:37.726 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:13:44.968 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:13:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:13:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:45.247 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:13:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:45.248 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:13:45.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-fuse-debuginfo 2023-10-30T12:13:45.993 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:09:35 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:13:46.142 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:13:46.143 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:46.143 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:13:46.143 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:46.143 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:13:46.144 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.8 M 2023-10-30T12:13:46.144 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:46.144 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:13:46.144 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:46.144 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:13:46.144 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:46.145 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 5.8 M 2023-10-30T12:13:46.145 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 28 M 2023-10-30T12:13:46.145 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:13:46.983 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x 6.9 MB/s | 5.8 MB 00:00 2023-10-30T12:13:46.984 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:46.984 INFO:teuthology.orchestra.run.smithi150.stdout:Total 6.9 MB/s | 5.8 MB 00:00 2023-10-30T12:13:46.985 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:13:46.988 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:13:46.988 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:13:47.056 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:13:47.057 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:13:47.774 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:13:47.874 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:13:48.081 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-mirror-debuginfo-18.0.0-6839.gcc36dd85.el9. 4.9 MB/s | 51 MB 00:10 2023-10-30T12:13:48.082 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:48.082 INFO:teuthology.orchestra.run.smithi148.stdout:Total 4.9 MB/s | 51 MB 00:10 2023-10-30T12:13:48.082 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:13:48.086 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:13:48.086 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:13:48.420 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:13:48.420 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:13:48.478 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:13:48.755 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:13:48.756 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:48.756 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:13:48.756 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:48.756 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:48.757 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:13:48.907 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados-devel-debuginfo 2023-10-30T12:13:49.455 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:09:39 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:13:49.596 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:13:49.597 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:49.597 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:13:49.597 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:49.597 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:13:49.598 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 108 k 2023-10-30T12:13:49.598 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:49.598 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:13:49.598 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:49.598 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:13:49.599 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:49.599 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 108 k 2023-10-30T12:13:49.599 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 440 k 2023-10-30T12:13:49.599 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:13:50.136 INFO:teuthology.orchestra.run.smithi150.stdout:librados-devel-debuginfo-18.0.0-6839.gcc36dd85. 202 kB/s | 108 kB 00:00 2023-10-30T12:13:50.137 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:50.137 INFO:teuthology.orchestra.run.smithi150.stdout:Total 201 kB/s | 108 kB 00:00 2023-10-30T12:13:50.137 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:13:50.140 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:13:50.140 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:13:50.174 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:13:50.174 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:13:50.290 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:13:50.392 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:13:50.745 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:13:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:13:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:51.062 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:13:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:51.062 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:51.062 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:13:51.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs2-debuginfo 2023-10-30T12:13:51.767 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:09:41 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:13:51.908 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:13:51.909 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:51.910 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:13:51.910 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:51.910 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:13:51.910 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.1 M 2023-10-30T12:13:51.910 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:51.911 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:13:51.911 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:51.911 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:13:51.911 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:51.911 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 5.1 M 2023-10-30T12:13:51.912 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 24 M 2023-10-30T12:13:51.912 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:13:54.227 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:13:54.645 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:13:54.780 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs2-debuginfo-18.0.0-6839.gcc36dd85.el9. 1.8 MB/s | 5.1 MB 00:02 2023-10-30T12:13:54.781 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:54.781 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.8 MB/s | 5.1 MB 00:02 2023-10-30T12:13:54.782 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:13:54.786 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:13:54.786 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:13:54.852 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:13:54.853 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:13:55.492 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:13:55.603 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:13:56.137 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:13:56.421 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:13:56.422 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:56.422 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:13:56.422 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:56.422 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:56.422 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:13:56.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados2-debuginfo 2023-10-30T12:13:56.647 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:13:56.920 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:13:56.920 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:56.921 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:13:56.921 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:13:56.921 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:56.921 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:13:57.112 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-nbd-debuginfo 2023-10-30T12:13:57.128 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:09:47 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:13:57.268 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:13:57.269 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:57.269 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:13:57.270 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:57.270 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:13:57.270 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 M 2023-10-30T12:13:57.270 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:57.270 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:13:57.271 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:13:57.271 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:13:57.271 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:13:57.271 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 45 M 2023-10-30T12:13:57.271 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 255 M 2023-10-30T12:13:57.272 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:13:57.658 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:10:09 ago on Mon 30 Oct 2023 12:03:48 PM UTC. 2023-10-30T12:13:57.799 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-30T12:13:57.800 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:57.800 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2023-10-30T12:13:57.801 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:57.801 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-10-30T12:13:57.801 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.2 M 2023-10-30T12:13:57.801 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:57.801 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-10-30T12:13:57.802 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-10-30T12:13:57.802 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2023-10-30T12:13:57.802 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:13:57.802 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 1.2 M 2023-10-30T12:13:57.802 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 5.4 M 2023-10-30T12:13:57.803 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-10-30T12:13:59.540 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-nbd-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 725 kB/s | 1.2 MB 00:01 2023-10-30T12:13:59.541 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:13:59.542 INFO:teuthology.orchestra.run.smithi148.stdout:Total 724 kB/s | 1.2 MB 00:01 2023-10-30T12:13:59.542 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-10-30T12:13:59.545 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-10-30T12:13:59.546 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-10-30T12:13:59.590 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-10-30T12:13:59.591 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-10-30T12:13:59.840 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-10-30T12:13:59.956 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:14:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:14:00.719 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:14:00.719 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:14:00.719 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-10-30T12:14:00.720 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:14:00.720 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:14:00.720 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-30T12:14:00.904 DEBUG:teuthology.parallel:result is None 2023-10-30T12:14:12.007 INFO:teuthology.orchestra.run.smithi150.stdout:librados2-debuginfo-18.0.0-6839.gcc36dd85.el9.x 3.1 MB/s | 45 MB 00:14 2023-10-30T12:14:12.007 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:14:12.008 INFO:teuthology.orchestra.run.smithi150.stdout:Total 3.1 MB/s | 45 MB 00:14 2023-10-30T12:14:12.008 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:14:12.012 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:14:12.012 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:14:12.324 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:14:12.325 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:14:17.479 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:14:17.588 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:14:19.448 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:14:19.716 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:14:19.716 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:14:19.716 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:14:19.717 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:14:19.717 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:14:19.717 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:14:19.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librbd1-debuginfo 2023-10-30T12:14:20.425 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:10:10 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:14:20.565 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:14:20.565 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:14:20.566 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:14:20.566 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:14:20.566 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:14:20.566 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T12:14:20.567 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:14:20.567 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:14:20.567 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:14:20.567 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:14:20.567 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:14:20.568 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 51 M 2023-10-30T12:14:20.568 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 298 M 2023-10-30T12:14:20.568 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:14:48.123 INFO:teuthology.orchestra.run.smithi150.stdout:librbd1-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 1.8 MB/s | 51 MB 00:27 2023-10-30T12:14:48.125 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:14:48.126 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.8 MB/s | 51 MB 00:27 2023-10-30T12:14:48.126 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:14:48.128 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:14:48.129 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:14:48.473 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:14:48.473 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:14:54.328 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:14:54.606 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:14:56.720 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:14:57.011 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:14:57.011 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:14:57.011 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:14:57.011 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:14:57.012 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:14:57.012 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:14:57.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-cephfs-debuginfo 2023-10-30T12:14:57.734 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:10:47 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:14:57.874 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:14:57.875 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:14:57.875 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:14:57.875 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:14:57.876 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:14:57.876 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 615 k 2023-10-30T12:14:57.876 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:14:57.876 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:14:57.876 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:14:57.877 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:14:57.877 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:14:57.877 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 615 k 2023-10-30T12:14:57.877 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 2.1 M 2023-10-30T12:14:57.877 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:14:58.417 INFO:teuthology.orchestra.run.smithi150.stdout:python3-cephfs-debuginfo-18.0.0-6839.gcc36dd85. 1.1 MB/s | 615 kB 00:00 2023-10-30T12:14:58.418 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:14:58.418 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.1 MB/s | 615 kB 00:00 2023-10-30T12:14:58.419 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:14:58.421 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:14:58.422 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:14:58.458 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:14:58.459 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:14:58.621 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:14:58.732 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:14:59.175 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:14:59.452 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:14:59.452 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:14:59.452 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:14:59.453 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:14:59.453 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:14:59.453 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:14:59.648 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rados-debuginfo 2023-10-30T12:15:00.195 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:10:50 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:15:00.338 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:15:00.338 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:00.338 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:15:00.338 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:00.339 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:15:00.339 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 770 k 2023-10-30T12:15:00.339 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:00.339 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:15:00.339 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:00.340 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:15:00.340 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:00.340 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 770 k 2023-10-30T12:15:00.340 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 2.7 M 2023-10-30T12:15:00.340 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:15:01.570 INFO:teuthology.orchestra.run.smithi150.stdout:python3-rados-debuginfo-18.0.0-6839.gcc36dd85.e 624 kB/s | 770 kB 00:01 2023-10-30T12:15:01.571 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:15:01.572 INFO:teuthology.orchestra.run.smithi150.stdout:Total 624 kB/s | 770 kB 00:01 2023-10-30T12:15:01.572 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:15:01.575 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:15:01.576 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:15:01.616 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:15:01.616 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:15:01.797 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:15:01.893 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:15:02.266 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:15:02.526 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:15:02.527 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:02.527 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:15:02.527 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:15:02.527 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:02.527 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:15:02.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rbd-debuginfo 2023-10-30T12:15:03.229 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:10:53 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:15:03.369 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:15:03.370 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:03.371 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:15:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:15:03.371 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.1 M 2023-10-30T12:15:03.372 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:15:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:15:03.372 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 1.1 M 2023-10-30T12:15:03.373 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 4.3 M 2023-10-30T12:15:03.373 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:15:05.023 INFO:teuthology.orchestra.run.smithi150.stdout:python3-rbd-debuginfo-18.0.0-6839.gcc36dd85.el9 697 kB/s | 1.1 MB 00:01 2023-10-30T12:15:05.024 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:15:05.024 INFO:teuthology.orchestra.run.smithi150.stdout:Total 696 kB/s | 1.1 MB 00:01 2023-10-30T12:15:05.025 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:15:05.028 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:15:05.029 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:15:05.072 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:15:05.073 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:15:05.310 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:15:05.421 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:15:05.802 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:15:06.058 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:15:06.059 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:06.059 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:15:06.059 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:15:06.059 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:06.059 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:15:06.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rgw-debuginfo 2023-10-30T12:15:06.765 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:10:56 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:15:06.905 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:15:06.906 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:06.906 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:15:06.907 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:06.907 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:15:06.907 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 337 k 2023-10-30T12:15:06.907 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:06.907 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:15:06.908 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:06.908 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:15:06.908 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:06.908 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 337 k 2023-10-30T12:15:06.908 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 1.0 M 2023-10-30T12:15:06.908 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:15:07.476 INFO:teuthology.orchestra.run.smithi150.stdout:python3-rgw-debuginfo-18.0.0-6839.gcc36dd85.el9 592 kB/s | 337 kB 00:00 2023-10-30T12:15:07.477 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:15:07.477 INFO:teuthology.orchestra.run.smithi150.stdout:Total 591 kB/s | 337 kB 00:00 2023-10-30T12:15:07.478 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:15:07.481 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:15:07.481 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:15:07.518 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:15:07.518 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:15:07.653 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:15:07.750 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:15:08.106 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:15:08.376 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:15:08.376 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:08.377 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:15:08.377 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:15:08.377 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:08.377 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:15:08.533 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-fuse-debuginfo 2023-10-30T12:15:09.081 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:10:59 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:15:09.221 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:15:09.246 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:09.247 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:15:09.247 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:09.247 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:15:09.247 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 506 k 2023-10-30T12:15:09.247 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:09.247 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:15:09.248 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:09.248 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:15:09.248 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:09.248 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 506 k 2023-10-30T12:15:09.248 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 2.5 M 2023-10-30T12:15:09.249 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:15:10.296 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 472 kB/s | 506 kB 00:01 2023-10-30T12:15:10.296 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:15:10.297 INFO:teuthology.orchestra.run.smithi150.stdout:Total 471 kB/s | 506 kB 00:01 2023-10-30T12:15:10.297 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:15:10.300 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:15:10.300 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:15:10.338 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:15:10.338 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:15:10.499 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:15:10.594 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:15:10.985 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:15:11.242 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:15:11.243 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:11.243 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:15:11.243 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:15:11.243 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:11.243 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:15:11.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-mirror-debuginfo 2023-10-30T12:15:11.953 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:11:01 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:15:12.093 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:15:12.094 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:12.094 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:15:12.095 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:12.095 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:15:12.095 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T12:15:12.095 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:12.095 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:15:12.096 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:15:12.096 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:15:12.096 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:15:12.096 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 51 M 2023-10-30T12:15:12.096 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 289 M 2023-10-30T12:15:12.097 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:15:55.878 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-mirror-debuginfo-18.0.0-6839.gcc36dd85.el9. 1.2 MB/s | 51 MB 00:43 2023-10-30T12:15:55.879 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:15:55.879 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.2 MB/s | 51 MB 00:43 2023-10-30T12:15:55.879 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:15:55.883 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:15:55.883 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:15:56.215 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:15:56.216 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:16:02.003 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:16:02.434 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:16:04.431 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:16:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:16:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:04.711 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:16:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:16:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:04.712 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:16:04.858 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-nbd-debuginfo 2023-10-30T12:16:05.403 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:11:55 ago on Mon 30 Oct 2023 12:04:10 PM UTC. 2023-10-30T12:16:05.542 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-30T12:16:05.543 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:16:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-10-30T12:16:05.543 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:16:05.543 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-30T12:16:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.2 M 2023-10-30T12:16:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:05.544 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-30T12:16:05.544 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-30T12:16:05.544 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-10-30T12:16:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:05.545 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 1.2 M 2023-10-30T12:16:05.545 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 5.4 M 2023-10-30T12:16:05.545 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-30T12:16:07.113 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-nbd-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 803 kB/s | 1.2 MB 00:01 2023-10-30T12:16:07.114 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:16:07.114 INFO:teuthology.orchestra.run.smithi150.stdout:Total 802 kB/s | 1.2 MB 00:01 2023-10-30T12:16:07.114 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-30T12:16:07.118 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-30T12:16:07.118 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-30T12:16:07.163 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-30T12:16:07.163 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-30T12:16:07.412 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-30T12:16:07.511 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:16:07.935 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:16:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:16:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:08.218 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-30T12:16:08.218 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:16:08.219 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:08.219 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-30T12:16:08.368 DEBUG:teuthology.parallel:result is None 2023-10-30T12:16:08.369 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T12:16:08.659 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-30T12:16:08.683 INFO:teuthology.orchestra.run.smithi148.stdout:18.0.0-6839.gcc36dd85.el9 2023-10-30T12:16:08.684 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6839.gcc36dd85.el9 2023-10-30T12:16:08.684 INFO:teuthology.task.install:The correct ceph version 18.0.0-6839.gcc36dd85 is installed. 2023-10-30T12:16:08.685 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T12:16:08.912 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-30T12:16:08.934 INFO:teuthology.orchestra.run.smithi150.stdout:18.0.0-6839.gcc36dd85.el9 2023-10-30T12:16:08.935 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6839.gcc36dd85.el9 2023-10-30T12:16:08.935 INFO:teuthology.task.install:The correct ceph version 18.0.0-6839.gcc36dd85 is installed. 2023-10-30T12:16:08.935 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T12:16:08.936 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:08.936 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T12:16:08.968 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:08.968 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T12:16:09.002 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T12:16:09.003 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:09.003 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T12:16:09.037 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T12:16:09.106 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:09.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T12:16:09.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T12:16:09.202 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T12:16:09.203 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:09.203 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T12:16:09.232 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T12:16:09.301 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:09.301 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T12:16:09.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T12:16:09.396 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T12:16:09.396 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:09.396 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T12:16:09.424 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T12:16:09.492 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:09.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T12:16:09.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T12:16:09.585 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T12:16:09.648 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T12:16:09.649 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /var/log/ceph 2023-10-30T12:16:09.652 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2023-10-30T12:16:09.678 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T12:16:09.679 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T12:16:09.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T12:16:09.746 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T12:16:09.746 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T12:16:09.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T12:16:09.824 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T12:16:09.824 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T12:16:09.824 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7441076', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'd0db9f2f95b9d6344950185071cace2a50d296eb'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1876, 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'sleep_before_teardown': 0, 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd0db9f2f95b9d6344950185071cace2a50d296eb', 'targets': {'smithi148.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErD+fKORMbAS/4SGdN1qk+fU5KuhzMeu5Q6Ym1h8GMQsH5+45LOEyN9dgz8ltlxPMWIuTBvuay8FM/2eK7wFqM=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKgJDD52JQV1IuyEaPL+QAJj7lMW3qMUl3NiI2iKvVkDuf8z8E/576IK4HXp620zEqzKzeEFUb7IQIoHyd7zmSY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-30_10:36:20', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T12:16:09.825 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T12:16:09.856 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T12:16:09.871 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T12:16:09.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T12:16:09.939 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:09.939 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T12:16:09.992 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T12:16:09.992 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 2023-10-30T12:16:10.047 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T12:16:10.047 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:16:10.047 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-10-30T12:16:10.047 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:16:10.048 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:16:10.048 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-10-30 12:05:42.334564725 +0000 2023-10-30T12:16:10.048 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-10-30 12:02:53.314781677 +0000 2023-10-30T12:16:10.048 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-10-30 12:02:53.314781677 +0000 2023-10-30T12:16:10.048 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2023-10-30 12:02:53.314781677 +0000 2023-10-30T12:16:10.049 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T12:16:10.112 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-10-30T12:16:10.112 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-10-30T12:16:10.113 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000171987 s, 3.0 MB/s 2023-10-30T12:16:10.114 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T12:16:10.170 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 2023-10-30T12:16:10.224 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T12:16:10.224 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:16:10.225 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-10-30T12:16:10.225 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:16:10.225 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:16:10.225 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-10-30 12:05:42.335564701 +0000 2023-10-30T12:16:10.225 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-10-30 12:02:53.895768161 +0000 2023-10-30T12:16:10.225 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-10-30 12:02:53.895768161 +0000 2023-10-30T12:16:10.226 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2023-10-30 12:02:53.895768161 +0000 2023-10-30T12:16:10.226 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T12:16:10.289 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-10-30T12:16:10.289 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-10-30T12:16:10.290 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000172982 s, 3.0 MB/s 2023-10-30T12:16:10.291 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T12:16:10.346 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 2023-10-30T12:16:10.401 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T12:16:10.401 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:16:10.401 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-10-30T12:16:10.401 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:16:10.402 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:16:10.402 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-10-30 12:05:42.335564701 +0000 2023-10-30T12:16:10.402 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-10-30 12:02:54.383756807 +0000 2023-10-30T12:16:10.402 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-10-30 12:02:54.383756807 +0000 2023-10-30T12:16:10.403 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2023-10-30 12:02:54.383756807 +0000 2023-10-30T12:16:10.403 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T12:16:10.466 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-10-30T12:16:10.467 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-10-30T12:16:10.467 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000218277 s, 2.3 MB/s 2023-10-30T12:16:10.468 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T12:16:10.523 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 2023-10-30T12:16:10.577 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T12:16:10.577 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:16:10.578 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-10-30T12:16:10.578 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:16:10.578 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:16:10.578 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-10-30 12:05:42.335564701 +0000 2023-10-30T12:16:10.578 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-10-30 12:02:54.892744965 +0000 2023-10-30T12:16:10.579 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-10-30 12:02:54.892744965 +0000 2023-10-30T12:16:10.579 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2023-10-30 12:02:54.892744965 +0000 2023-10-30T12:16:10.579 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T12:16:10.643 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-10-30T12:16:10.685 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-10-30T12:16:10.686 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000204611 s, 2.5 MB/s 2023-10-30T12:16:10.687 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T12:16:10.742 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:16:10.743 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:10.743 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T12:16:10.762 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T12:16:10.762 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2023-10-30T12:16:10.819 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T12:16:10.820 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:16:10.820 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-10-30T12:16:10.820 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:16:10.820 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:16:10.820 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-30 12:06:09.862715308 +0000 2023-10-30T12:16:10.821 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-30 12:02:55.321052509 +0000 2023-10-30T12:16:10.821 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-30 12:02:55.321052509 +0000 2023-10-30T12:16:10.821 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2023-10-30 12:02:55.321052509 +0000 2023-10-30T12:16:10.821 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T12:16:10.889 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-30T12:16:10.889 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-30T12:16:10.890 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00021565 s, 2.4 MB/s 2023-10-30T12:16:10.891 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T12:16:10.948 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2023-10-30T12:16:11.006 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T12:16:11.006 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:16:11.006 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-10-30T12:16:11.006 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:16:11.006 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:16:11.007 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-30 12:06:09.863715289 +0000 2023-10-30T12:16:11.007 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-30 12:02:55.808043334 +0000 2023-10-30T12:16:11.007 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-30 12:02:55.808043334 +0000 2023-10-30T12:16:11.007 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2023-10-30 12:02:55.808043334 +0000 2023-10-30T12:16:11.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T12:16:11.074 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-30T12:16:11.074 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-30T12:16:11.075 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000259129 s, 2.0 MB/s 2023-10-30T12:16:11.076 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T12:16:11.133 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2023-10-30T12:16:11.189 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T12:16:11.189 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:16:11.189 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-10-30T12:16:11.189 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:16:11.190 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:16:11.190 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-30 12:06:09.863715289 +0000 2023-10-30T12:16:11.190 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-30 12:02:56.307033933 +0000 2023-10-30T12:16:11.190 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-30 12:02:56.307033933 +0000 2023-10-30T12:16:11.190 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2023-10-30 12:02:56.307033933 +0000 2023-10-30T12:16:11.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T12:16:11.256 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-30T12:16:11.256 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-30T12:16:11.256 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000249981 s, 2.0 MB/s 2023-10-30T12:16:11.257 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T12:16:11.313 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2023-10-30T12:16:11.368 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T12:16:11.369 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:16:11.369 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-10-30T12:16:11.369 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:16:11.369 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:16:11.369 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-30 12:06:09.863715289 +0000 2023-10-30T12:16:11.370 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-30 12:02:56.776025098 +0000 2023-10-30T12:16:11.370 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-30 12:02:56.776025098 +0000 2023-10-30T12:16:11.370 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2023-10-30 12:02:56.776025098 +0000 2023-10-30T12:16:11.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T12:16:11.435 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-30T12:16:11.435 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-30T12:16:11.435 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000208426 s, 2.5 MB/s 2023-10-30T12:16:11.436 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T12:16:11.493 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:16:11.493 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T12:16:11.494 INFO:tasks.ceph:Generating config... 2023-10-30T12:16:11.495 INFO:tasks.ceph:[client] debug monc = 20 2023-10-30T12:16:11.495 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T12:16:11.496 INFO:tasks.ceph:[client] debug objecter = 20 2023-10-30T12:16:11.496 INFO:tasks.ceph:[client] debug rados = 20 2023-10-30T12:16:11.496 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-30T12:16:11.496 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-10-30T12:16:11.497 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-10-30T12:16:11.497 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-10-30T12:16:11.497 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-30T12:16:11.498 INFO:tasks.ceph:[global] ms type = async 2023-10-30T12:16:11.498 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-10-30T12:16:11.498 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T12:16:11.498 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T12:16:11.499 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T12:16:11.499 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T12:16:11.499 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T12:16:11.499 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-10-30T12:16:11.500 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-10-30T12:16:11.500 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-10-30T12:16:11.500 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-10-30T12:16:11.500 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-10-30T12:16:11.501 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-10-30T12:16:11.501 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-10-30T12:16:11.501 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-10-30T12:16:11.501 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-30T12:16:11.502 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-30T12:16:11.502 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-10-30T12:16:11.502 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T12:16:11.503 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T12:16:11.503 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-10-30T12:16:11.503 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-10-30T12:16:11.503 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-10-30T12:16:11.504 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-30T12:16:11.504 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T12:16:11.504 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T12:16:11.504 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-30T12:16:11.505 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T12:16:11.505 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T12:16:11.505 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T12:16:11.505 INFO:tasks.ceph:[osd] osd class default list = * 2023-10-30T12:16:11.506 INFO:tasks.ceph:[osd] osd class load list = * 2023-10-30T12:16:11.506 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-10-30T12:16:11.506 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-10-30T12:16:11.506 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-10-30T12:16:11.507 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T12:16:11.507 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-10-30T12:16:11.507 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-10-30T12:16:11.507 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-10-30T12:16:11.508 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T12:16:11.508 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-10-30T12:16:11.508 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-10-30T12:16:11.508 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-30T12:16:11.509 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T12:16:11.509 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T12:16:11.770 INFO:teuthology.orchestra.run.smithi148.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T12:16:11.773 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T12:16:11.821 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:16:11.849 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.148'), ('mon.c', '[v2:172.21.15.148:3301,v1:172.21.15.148:6790]'), ('mon.b', '172.21.15.150')] 2023-10-30T12:16:11.849 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.148,[v2:172.21.15.148:3301,v1:172.21.15.148:6790],172.21.15.150', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-10-30T12:16:11.851 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:11.851 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T12:16:11.904 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.148 --addv c '[v2:172.21.15.148:3301,v1:172.21.15.148:6790]' --add b 172.21.15.150 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:11.989 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:11.989 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: generated fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:16:11.989 INFO:teuthology.orchestra.run.smithi148.stdout:setting min_mon_release = pacific 2023-10-30T12:16:11.989 INFO:teuthology.orchestra.run.smithi148.stdout:epoch 0 2023-10-30T12:16:11.990 INFO:teuthology.orchestra.run.smithi148.stdout:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:16:11.990 INFO:teuthology.orchestra.run.smithi148.stdout:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:16:11.990 INFO:teuthology.orchestra.run.smithi148.stdout:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:16:11.990 INFO:teuthology.orchestra.run.smithi148.stdout:min_mon_release 16 (pacific) 2023-10-30T12:16:11.990 INFO:teuthology.orchestra.run.smithi148.stdout:election_strategy: 3 2023-10-30T12:16:11.991 INFO:teuthology.orchestra.run.smithi148.stdout:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:16:11.991 INFO:teuthology.orchestra.run.smithi148.stdout:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:16:11.991 INFO:teuthology.orchestra.run.smithi148.stdout:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:16:11.991 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T12:16:11.992 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T12:16:12.007 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e2fdab2f-1adf-4b35-9eee-e40114aa8bbb... 2023-10-30T12:16:12.008 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T12:16:12.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T12:16:12.097 INFO:teuthology.orchestra.run.smithi148.stdout:[global] 2023-10-30T12:16:12.097 INFO:teuthology.orchestra.run.smithi148.stdout: chdir = "" 2023-10-30T12:16:12.097 INFO:teuthology.orchestra.run.smithi148.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T12:16:12.098 INFO:teuthology.orchestra.run.smithi148.stdout: auth supported = cephx 2023-10-30T12:16:12.098 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.098 INFO:teuthology.orchestra.run.smithi148.stdout: filestore xattr use omap = true 2023-10-30T12:16:12.098 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.098 INFO:teuthology.orchestra.run.smithi148.stdout: mon clock drift allowed = 1.000 2023-10-30T12:16:12.099 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.099 INFO:teuthology.orchestra.run.smithi148.stdout: osd crush chooseleaf type = 0 2023-10-30T12:16:12.099 INFO:teuthology.orchestra.run.smithi148.stdout: auth debug = true 2023-10-30T12:16:12.099 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.099 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on old message = true 2023-10-30T12:16:12.099 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on bug = true 2023-10-30T12:16:12.100 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.100 INFO:teuthology.orchestra.run.smithi148.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T12:16:12.100 INFO:teuthology.orchestra.run.smithi148.stdout: mon pg warn max object skew = 0 2023-10-30T12:16:12.100 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.100 INFO:teuthology.orchestra.run.smithi148.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T12:16:12.100 INFO:teuthology.orchestra.run.smithi148.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T12:16:12.101 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.101 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default size = 2 2023-10-30T12:16:12.101 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.101 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow primary affinity = true 2023-10-30T12:16:12.101 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow pg remap = true 2023-10-30T12:16:12.101 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on legacy crush tunables = false 2023-10-30T12:16:12.102 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on crush straw calc version zero = false 2023-10-30T12:16:12.102 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on no sortbitwise = false 2023-10-30T12:16:12.102 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on osd down out interval zero = false 2023-10-30T12:16:12.102 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on too few osds = false 2023-10-30T12:16:12.102 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T12:16:12.103 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T12:16:12.103 INFO:teuthology.orchestra.run.smithi148.stdout: mon_allow_pool_size_one = true 2023-10-30T12:16:12.103 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.103 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T12:16:12.103 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.103 INFO:teuthology.orchestra.run.smithi148.stdout: osd default data pool replay window = 5 2023-10-30T12:16:12.104 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.104 INFO:teuthology.orchestra.run.smithi148.stdout: mon allow pool delete = true 2023-10-30T12:16:12.104 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.104 INFO:teuthology.orchestra.run.smithi148.stdout: mon cluster log file level = debug 2023-10-30T12:16:12.104 INFO:teuthology.orchestra.run.smithi148.stdout: debug asserts on shutdown = true 2023-10-30T12:16:12.104 INFO:teuthology.orchestra.run.smithi148.stdout: mon health detail to clog = false 2023-10-30T12:16:12.105 INFO:teuthology.orchestra.run.smithi148.stdout: mon host = "172.21.15.148,[v2:172.21.15.148:3301,v1:172.21.15.148:6790],172.21.15.150" 2023-10-30T12:16:12.105 INFO:teuthology.orchestra.run.smithi148.stdout: mon client directed command retry = 5 2023-10-30T12:16:12.105 INFO:teuthology.orchestra.run.smithi148.stdout: mon election default strategy = 3 2023-10-30T12:16:12.105 INFO:teuthology.orchestra.run.smithi148.stdout: ms bind msgr1 = False 2023-10-30T12:16:12.105 INFO:teuthology.orchestra.run.smithi148.stdout: ms bind msgr2 = True 2023-10-30T12:16:12.105 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject socket failures = 5000 2023-10-30T12:16:12.106 INFO:teuthology.orchestra.run.smithi148.stdout: ms type = async 2023-10-30T12:16:12.106 INFO:teuthology.orchestra.run.smithi148.stdout: osd heartbeat grace = 80 2023-10-30T12:16:12.106 INFO:teuthology.orchestra.run.smithi148.stdout: fsid = e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:16:12.106 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.106 INFO:teuthology.orchestra.run.smithi148.stdout:[osd] 2023-10-30T12:16:12.107 INFO:teuthology.orchestra.run.smithi148.stdout: osd journal size = 100 2023-10-30T12:16:12.107 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.107 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub load threshold = 5.0 2023-10-30T12:16:12.107 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub max interval = 600 2023-10-30T12:16:12.107 INFO:teuthology.orchestra.run.smithi148.stdout: osd mclock profile = high_recovery_ops 2023-10-30T12:16:12.107 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.108 INFO:teuthology.orchestra.run.smithi148.stdout: osd recover clone overlap = true 2023-10-30T12:16:12.108 INFO:teuthology.orchestra.run.smithi148.stdout: osd recovery max chunk = 1048576 2023-10-30T12:16:12.108 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.108 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug shutdown = true 2023-10-30T12:16:12.108 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug op order = true 2023-10-30T12:16:12.108 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify stray on activate = true 2023-10-30T12:16:12.109 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.109 INFO:teuthology.orchestra.run.smithi148.stdout: osd open classes on start = true 2023-10-30T12:16:12.109 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug pg log writeout = true 2023-10-30T12:16:12.109 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.109 INFO:teuthology.orchestra.run.smithi148.stdout: osd deep scrub update digest min age = 30 2023-10-30T12:16:12.109 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.110 INFO:teuthology.orchestra.run.smithi148.stdout: osd map max advance = 10 2023-10-30T12:16:12.110 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.110 INFO:teuthology.orchestra.run.smithi148.stdout: journal zero on create = true 2023-10-30T12:16:12.110 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.110 INFO:teuthology.orchestra.run.smithi148.stdout: filestore ondisk finisher threads = 3 2023-10-30T12:16:12.110 INFO:teuthology.orchestra.run.smithi148.stdout: filestore apply finisher threads = 3 2023-10-30T12:16:12.111 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.111 INFO:teuthology.orchestra.run.smithi148.stdout: bdev debug aio = true 2023-10-30T12:16:12.111 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug misdirected ops = true 2023-10-30T12:16:12.111 INFO:teuthology.orchestra.run.smithi148.stdout: bdev async discard = True 2023-10-30T12:16:12.111 INFO:teuthology.orchestra.run.smithi148.stdout: bdev enable discard = True 2023-10-30T12:16:12.112 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore allocator = stupid 2023-10-30T12:16:12.112 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore block size = 96636764160 2023-10-30T12:16:12.112 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore fsck on mount = True 2023-10-30T12:16:12.112 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore zero block detection = True 2023-10-30T12:16:12.112 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluefs = 1 2023-10-30T12:16:12.112 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluestore = 1 2023-10-30T12:16:12.112 INFO:teuthology.orchestra.run.smithi148.stdout: debug monc = 20 2023-10-30T12:16:12.113 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2023-10-30T12:16:12.113 INFO:teuthology.orchestra.run.smithi148.stdout: debug osd = 20 2023-10-30T12:16:12.113 INFO:teuthology.orchestra.run.smithi148.stdout: debug rocksdb = 10 2023-10-30T12:16:12.113 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T12:16:12.113 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd full ratio = 0.9 2023-10-30T12:16:12.114 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd nearfull ratio = 0.8 2023-10-30T12:16:12.114 INFO:teuthology.orchestra.run.smithi148.stdout: osd class default list = * 2023-10-30T12:16:12.114 INFO:teuthology.orchestra.run.smithi148.stdout: osd class load list = * 2023-10-30T12:16:12.114 INFO:teuthology.orchestra.run.smithi148.stdout: osd client watch timeout = 120 2023-10-30T12:16:12.114 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify cached snaps = True 2023-10-30T12:16:12.114 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify missing on start = True 2023-10-30T12:16:12.115 INFO:teuthology.orchestra.run.smithi148.stdout: osd failsafe full ratio = 0.95 2023-10-30T12:16:12.115 INFO:teuthology.orchestra.run.smithi148.stdout: osd fast shutdown = False 2023-10-30T12:16:12.115 INFO:teuthology.orchestra.run.smithi148.stdout: osd mclock override recovery settings = True 2023-10-30T12:16:12.115 INFO:teuthology.orchestra.run.smithi148.stdout: osd objectstore = bluestore 2023-10-30T12:16:12.115 INFO:teuthology.orchestra.run.smithi148.stdout: osd op queue = debug_random 2023-10-30T12:16:12.115 INFO:teuthology.orchestra.run.smithi148.stdout: osd op queue cut off = debug_random 2023-10-30T12:16:12.116 INFO:teuthology.orchestra.run.smithi148.stdout: osd shutdown pgref assert = True 2023-10-30T12:16:12.116 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.116 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr] 2023-10-30T12:16:12.116 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2023-10-30T12:16:12.116 INFO:teuthology.orchestra.run.smithi148.stdout: debug mgr = 20 2023-10-30T12:16:12.116 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2023-10-30T12:16:12.117 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2023-10-30T12:16:12.117 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:16:12.117 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:16:12.117 INFO:teuthology.orchestra.run.smithi148.stdout: mgr/telemetry/nag = false 2023-10-30T12:16:12.117 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.117 INFO:teuthology.orchestra.run.smithi148.stdout:[mon] 2023-10-30T12:16:12.118 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2023-10-30T12:16:12.118 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2023-10-30T12:16:12.118 INFO:teuthology.orchestra.run.smithi148.stdout: debug paxos = 20 2023-10-30T12:16:12.118 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2023-10-30T12:16:12.118 INFO:teuthology.orchestra.run.smithi148.stdout: mon data avail warn = 5 2023-10-30T12:16:12.119 INFO:teuthology.orchestra.run.smithi148.stdout: mon mgr mkfs grace = 240 2023-10-30T12:16:12.119 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:16:12.119 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd reporter subtree level = osd 2023-10-30T12:16:12.119 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd prime pg temp = true 2023-10-30T12:16:12.119 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:16:12.119 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.120 INFO:teuthology.orchestra.run.smithi148.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T12:16:12.120 INFO:teuthology.orchestra.run.smithi148.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T12:16:12.120 INFO:teuthology.orchestra.run.smithi148.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T12:16:12.120 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.120 INFO:teuthology.orchestra.run.smithi148.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T12:16:12.120 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T12:16:12.121 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T12:16:12.121 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.121 INFO:teuthology.orchestra.run.smithi148.stdout: # 1m isn't quite enough 2023-10-30T12:16:12.121 INFO:teuthology.orchestra.run.smithi148.stdout: mon_down_mkfs_grace = 2m 2023-10-30T12:16:12.121 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.121 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_filestore_osds = false 2023-10-30T12:16:12.122 INFO:teuthology.orchestra.run.smithi148.stdout: mon min osdmap epochs = 50 2023-10-30T12:16:12.122 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd crush smoke test = False 2023-10-30T12:16:12.122 INFO:teuthology.orchestra.run.smithi148.stdout: mon osdmap full prune interval = 2 2023-10-30T12:16:12.122 INFO:teuthology.orchestra.run.smithi148.stdout: mon osdmap full prune min = 15 2023-10-30T12:16:12.122 INFO:teuthology.orchestra.run.smithi148.stdout: mon osdmap full prune txsize = 2 2023-10-30T12:16:12.122 INFO:teuthology.orchestra.run.smithi148.stdout: mon scrub interval = 300 2023-10-30T12:16:12.123 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on pool no app = False 2023-10-30T12:16:12.123 INFO:teuthology.orchestra.run.smithi148.stdout: paxos service trim min = 10 2023-10-30T12:16:12.123 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:16:12.123 INFO:teuthology.orchestra.run.smithi148.stdout:[client] 2023-10-30T12:16:12.123 INFO:teuthology.orchestra.run.smithi148.stdout: rgw cache enabled = true 2023-10-30T12:16:12.123 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable ops log = true 2023-10-30T12:16:12.124 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable usage log = true 2023-10-30T12:16:12.124 INFO:teuthology.orchestra.run.smithi148.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T12:16:12.124 INFO:teuthology.orchestra.run.smithi148.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T12:16:12.124 INFO:teuthology.orchestra.run.smithi148.stdout: debug monc = 20 2023-10-30T12:16:12.124 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2023-10-30T12:16:12.124 INFO:teuthology.orchestra.run.smithi148.stdout: debug objecter = 20 2023-10-30T12:16:12.125 INFO:teuthology.orchestra.run.smithi148.stdout: debug rados = 20 2023-10-30T12:16:12.125 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.a] 2023-10-30T12:16:12.125 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.c] 2023-10-30T12:16:12.125 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.b] 2023-10-30T12:16:12.126 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2023-10-30T12:16:12.127 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2023-10-30T12:16:12.127 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T12:16:12.127 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2023-10-30T12:16:12.127 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.127 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2023-10-30T12:16:12.128 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.128 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2023-10-30T12:16:12.128 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.128 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2023-10-30T12:16:12.128 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2023-10-30T12:16:12.129 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.129 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2023-10-30T12:16:12.129 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2023-10-30T12:16:12.129 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.129 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T12:16:12.129 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2023-10-30T12:16:12.130 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.130 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T12:16:12.130 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T12:16:12.130 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.130 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2023-10-30T12:16:12.130 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.131 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2023-10-30T12:16:12.131 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2023-10-30T12:16:12.131 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2023-10-30T12:16:12.131 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2023-10-30T12:16:12.131 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2023-10-30T12:16:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2023-10-30T12:16:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2023-10-30T12:16:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T12:16:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T12:16:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2023-10-30T12:16:12.133 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.133 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T12:16:12.133 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.133 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2023-10-30T12:16:12.133 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2023-10-30T12:16:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2023-10-30T12:16:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2023-10-30T12:16:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2023-10-30T12:16:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.148,[v2:172.21.15.148:3301,v1:172.21.15.148:6790],172.21.15.150" 2023-10-30T12:16:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: mon client directed command retry = 5 2023-10-30T12:16:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: mon election default strategy = 3 2023-10-30T12:16:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: ms bind msgr1 = False 2023-10-30T12:16:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: ms bind msgr2 = True 2023-10-30T12:16:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject socket failures = 5000 2023-10-30T12:16:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: ms type = async 2023-10-30T12:16:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: osd heartbeat grace = 80 2023-10-30T12:16:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:16:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.136 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2023-10-30T12:16:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2023-10-30T12:16:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2023-10-30T12:16:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2023-10-30T12:16:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2023-10-30T12:16:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2023-10-30T12:16:12.138 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2023-10-30T12:16:12.138 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.138 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2023-10-30T12:16:12.138 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2023-10-30T12:16:12.138 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2023-10-30T12:16:12.138 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2023-10-30T12:16:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2023-10-30T12:16:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2023-10-30T12:16:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2023-10-30T12:16:12.140 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.140 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2023-10-30T12:16:12.140 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.140 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2023-10-30T12:16:12.140 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2023-10-30T12:16:12.140 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.141 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2023-10-30T12:16:12.141 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2023-10-30T12:16:12.141 INFO:teuthology.orchestra.run.smithi150.stdout: bdev async discard = True 2023-10-30T12:16:12.141 INFO:teuthology.orchestra.run.smithi150.stdout: bdev enable discard = True 2023-10-30T12:16:12.141 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore allocator = stupid 2023-10-30T12:16:12.142 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2023-10-30T12:16:12.142 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2023-10-30T12:16:12.142 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore zero block detection = True 2023-10-30T12:16:12.142 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 1 2023-10-30T12:16:12.142 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 1 2023-10-30T12:16:12.143 INFO:teuthology.orchestra.run.smithi150.stdout: debug monc = 20 2023-10-30T12:16:12.143 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-30T12:16:12.143 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2023-10-30T12:16:12.143 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2023-10-30T12:16:12.143 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T12:16:12.144 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2023-10-30T12:16:12.144 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2023-10-30T12:16:12.144 INFO:teuthology.orchestra.run.smithi150.stdout: osd class default list = * 2023-10-30T12:16:12.144 INFO:teuthology.orchestra.run.smithi150.stdout: osd class load list = * 2023-10-30T12:16:12.144 INFO:teuthology.orchestra.run.smithi150.stdout: osd client watch timeout = 120 2023-10-30T12:16:12.144 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify cached snaps = True 2023-10-30T12:16:12.145 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify missing on start = True 2023-10-30T12:16:12.145 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2023-10-30T12:16:12.145 INFO:teuthology.orchestra.run.smithi150.stdout: osd fast shutdown = False 2023-10-30T12:16:12.145 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock override recovery settings = True 2023-10-30T12:16:12.145 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2023-10-30T12:16:12.146 INFO:teuthology.orchestra.run.smithi150.stdout: osd op queue = debug_random 2023-10-30T12:16:12.146 INFO:teuthology.orchestra.run.smithi150.stdout: osd op queue cut off = debug_random 2023-10-30T12:16:12.146 INFO:teuthology.orchestra.run.smithi150.stdout: osd shutdown pgref assert = True 2023-10-30T12:16:12.146 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.146 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2023-10-30T12:16:12.147 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-30T12:16:12.147 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2023-10-30T12:16:12.147 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2023-10-30T12:16:12.147 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2023-10-30T12:16:12.147 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:16:12.148 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:16:12.148 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2023-10-30T12:16:12.148 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.148 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2023-10-30T12:16:12.148 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-30T12:16:12.148 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2023-10-30T12:16:12.149 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2023-10-30T12:16:12.149 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2023-10-30T12:16:12.149 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2023-10-30T12:16:12.149 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2023-10-30T12:16:12.149 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:16:12.150 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2023-10-30T12:16:12.150 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2023-10-30T12:16:12.150 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:16:12.150 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.150 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T12:16:12.151 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T12:16:12.151 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T12:16:12.151 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.151 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T12:16:12.151 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T12:16:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T12:16:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2023-10-30T12:16:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2023-10-30T12:16:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2023-10-30T12:16:12.153 INFO:teuthology.orchestra.run.smithi150.stdout: mon min osdmap epochs = 50 2023-10-30T12:16:12.153 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd crush smoke test = False 2023-10-30T12:16:12.153 INFO:teuthology.orchestra.run.smithi150.stdout: mon osdmap full prune interval = 2 2023-10-30T12:16:12.153 INFO:teuthology.orchestra.run.smithi150.stdout: mon osdmap full prune min = 15 2023-10-30T12:16:12.153 INFO:teuthology.orchestra.run.smithi150.stdout: mon osdmap full prune txsize = 2 2023-10-30T12:16:12.154 INFO:teuthology.orchestra.run.smithi150.stdout: mon scrub interval = 300 2023-10-30T12:16:12.154 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on pool no app = False 2023-10-30T12:16:12.154 INFO:teuthology.orchestra.run.smithi150.stdout: paxos service trim min = 10 2023-10-30T12:16:12.154 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T12:16:12.154 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2023-10-30T12:16:12.155 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2023-10-30T12:16:12.155 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2023-10-30T12:16:12.155 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2023-10-30T12:16:12.155 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T12:16:12.155 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T12:16:12.156 INFO:teuthology.orchestra.run.smithi150.stdout: debug monc = 20 2023-10-30T12:16:12.156 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-30T12:16:12.156 INFO:teuthology.orchestra.run.smithi150.stdout: debug objecter = 20 2023-10-30T12:16:12.156 INFO:teuthology.orchestra.run.smithi150.stdout: debug rados = 20 2023-10-30T12:16:12.156 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2023-10-30T12:16:12.156 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2023-10-30T12:16:12.157 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2023-10-30T12:16:12.157 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T12:16:12.157 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T12:16:12.221 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T12:16:12.222 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:12.222 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T12:16:12.237 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:12.237 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T12:16:12.291 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:16:12.292 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:12.292 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T12:16:12.292 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:16:12.371 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:12.371 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:12.425 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:16:12.426 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:12.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T12:16:12.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:16:12.470 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:12.470 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:12.525 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T12:16:12.526 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T12:16:12.526 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T12:16:12.580 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T12:16:12.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T12:16:12.667 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T12:16:12.670 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T12:16:12.670 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T12:16:12.670 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T12:16:12.712 INFO:teuthology.orchestra.run.smithi148.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T12:16:12.728 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-30T12:16:12.770 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-30T12:16:12.786 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T12:16:12.786 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T12:16:12.786 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T12:16:12.813 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:16:12.813 INFO:tasks.ceph:role: osd.0 2023-10-30T12:16:12.813 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:16:12.814 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T12:16:12.887 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:16:12.888 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:16:12.888 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:16:12.888 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:16:12.888 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:16:12.889 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:16:12.889 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:16:12.889 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:16:12.889 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:16:12.889 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:16:13.401 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2023-10-30T12:16:13.403 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2023-10-30T12:16:13.403 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T12:16:13.514 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T12:16:13.550 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T12:16:13.617 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:16:13.617 INFO:tasks.ceph:role: osd.1 2023-10-30T12:16:13.618 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:16:13.618 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T12:16:13.691 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:16:13.691 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:16:13.691 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:16:13.692 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:16:13.692 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:16:13.692 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:16:13.692 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:16:13.692 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:16:13.692 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:16:13.693 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:16:14.149 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2023-10-30T12:16:14.151 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2023-10-30T12:16:14.151 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T12:16:14.243 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T12:16:14.275 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T12:16:14.343 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:16:14.344 INFO:tasks.ceph:role: osd.2 2023-10-30T12:16:14.344 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:16:14.344 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T12:16:14.419 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:16:14.419 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:16:14.419 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:16:14.420 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:16:14.420 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:16:14.420 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:16:14.420 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:16:14.420 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:16:14.420 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:16:14.421 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:16:14.916 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2023-10-30T12:16:14.919 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2023-10-30T12:16:14.919 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T12:16:15.068 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T12:16:15.106 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T12:16:15.177 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:16:15.177 INFO:tasks.ceph:role: osd.3 2023-10-30T12:16:15.178 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:16:15.178 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T12:16:15.253 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:16:15.254 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:16:15.254 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:16:15.254 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:16:15.254 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:16:15.255 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:16:15.255 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:16:15.255 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:16:15.255 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:16:15.255 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:16:15.766 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2023-10-30T12:16:15.768 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2023-10-30T12:16:15.768 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-30T12:16:15.889 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T12:16:15.924 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:16.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:16.020+0000 7f6f5321f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T12:16:16.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:16.021+0000 7f6f5321f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T12:16:16.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:16.021+0000 7f6f5321f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:16:16.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:16.021+0000 7f6f5321f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:16:16.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:16.021+0000 7f6f5321f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:16:16.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:16.021+0000 7f6f5321f500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T12:16:18.865 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T12:16:18.897 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:18.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:18.986+0000 7f4b48216500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T12:16:18.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:18.987+0000 7f4b48216500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T12:16:18.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:18.987+0000 7f4b48216500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:16:18.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:18.987+0000 7f4b48216500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:16:18.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:18.987+0000 7f4b48216500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:16:18.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:18.987+0000 7f4b48216500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T12:16:21.836 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T12:16:21.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:21.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:21.956+0000 7fa846a0f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T12:16:21.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:21.956+0000 7fa846a0f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T12:16:21.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:21.956+0000 7fa846a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:16:21.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:21.956+0000 7fa846a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:16:21.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:21.956+0000 7fa846a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:16:21.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:21.957+0000 7fa846a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T12:16:24.793 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T12:16:24.823 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:24.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:24.910+0000 7fe937907500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T12:16:24.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:24.911+0000 7fe937907500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T12:16:24.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:24.911+0000 7fe937907500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:16:24.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:24.911+0000 7fe937907500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:16:24.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:24.911+0000 7fe937907500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:16:24.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:24.911+0000 7fe937907500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T12:16:27.750 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T12:16:27.779 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T12:16:27.811 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:16:27.811 INFO:tasks.ceph:role: osd.4 2023-10-30T12:16:27.811 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:16:27.812 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T12:16:27.888 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:16:27.888 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:16:27.888 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:16:27.889 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:16:27.889 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:16:27.889 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:16:27.889 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:16:27.889 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:16:27.889 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:16:27.890 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:16:28.379 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-30T12:16:28.382 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-30T12:16:28.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-30T12:16:28.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T12:16:28.558 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T12:16:28.627 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:16:28.628 INFO:tasks.ceph:role: osd.5 2023-10-30T12:16:28.628 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:16:28.628 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T12:16:28.702 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:16:28.703 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:16:28.703 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:16:28.703 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:16:28.703 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:16:28.704 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:16:28.704 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:16:28.704 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:16:28.704 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:16:28.704 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:16:29.152 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-30T12:16:29.155 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-30T12:16:29.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-30T12:16:29.246 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T12:16:29.280 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-30T12:16:29.348 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:16:29.349 INFO:tasks.ceph:role: osd.6 2023-10-30T12:16:29.349 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:16:29.349 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T12:16:29.425 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:16:29.425 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:16:29.425 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:16:29.426 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:16:29.426 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:16:29.426 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:16:29.426 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:16:29.426 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:16:29.427 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:16:29.427 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:16:29.910 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-30T12:16:29.912 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-30T12:16:29.913 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-30T12:16:30.043 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-30T12:16:30.077 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-30T12:16:30.146 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:16:30.146 INFO:tasks.ceph:role: osd.7 2023-10-30T12:16:30.147 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:16:30.147 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T12:16:30.223 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:16:30.223 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:16:30.223 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:16:30.223 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:16:30.224 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:16:30.224 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:16:30.224 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:16:30.224 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:16:30.224 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:16:30.224 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:16:30.749 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-30T12:16:30.751 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-30T12:16:30.752 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-30T12:16:30.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-30T12:16:30.950 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:31.050 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:31.048+0000 7fa0b66d2500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T12:16:31.050 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:31.048+0000 7fa0b66d2500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T12:16:31.050 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:31.048+0000 7fa0b66d2500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:16:31.050 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:31.048+0000 7fa0b66d2500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:16:31.051 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:31.048+0000 7fa0b66d2500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:16:31.051 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:31.049+0000 7fa0b66d2500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T12:16:33.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T12:16:33.922 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:34.011 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:34.009+0000 7efc7f012500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T12:16:34.011 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:34.009+0000 7efc7f012500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T12:16:34.011 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:34.009+0000 7efc7f012500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:16:34.012 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:34.009+0000 7efc7f012500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:16:34.012 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:34.009+0000 7efc7f012500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:16:34.012 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:34.010+0000 7efc7f012500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T12:16:36.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T12:16:36.878 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:36.969 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:36.967+0000 7f114582b500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-30T12:16:36.969 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:36.968+0000 7f114582b500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-30T12:16:36.970 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:36.968+0000 7f114582b500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:16:36.970 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:36.968+0000 7f114582b500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:16:36.970 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:36.968+0000 7f114582b500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:16:36.971 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:36.969+0000 7f114582b500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-30T12:16:39.812 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-30T12:16:39.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:39.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:39.932+0000 7fcf2341c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-30T12:16:39.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:39.932+0000 7fcf2341c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-30T12:16:39.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:39.932+0000 7fcf2341c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:16:39.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:39.932+0000 7fcf2341c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:16:39.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:39.932+0000 7fcf2341c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:16:39.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:16:39.933+0000 7fcf2341c500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-30T12:16:42.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-30T12:16:42.801 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T12:16:42.802 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:42.802 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-30T12:16:42.833 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:42.834 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T12:16:42.904 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:42.904 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T12:16:42.974 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:42.974 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T12:16:43.043 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:43.044 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T12:16:43.112 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:43.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T12:16:43.144 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:43.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T12:16:43.214 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:43.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T12:16:43.283 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:43.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-30T12:16:43.354 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:43.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-30T12:16:43.424 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:16:43.424 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T12:16:43.442 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:16:43.443 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-30T12:16:43.480 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T12:16:43.481 DEBUG:teuthology.orchestra.run.smithi148:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T12:16:43.484 DEBUG:teuthology.orchestra.run.smithi150:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T12:16:43.523 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.y] 2023-10-30T12:16:43.523 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCMnj9lqD+aIhAAnyb0blX8YfsHIgb1Smgu8Q== 2023-10-30T12:16:43.523 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.0] 2023-10-30T12:16:43.524 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCQnj9l651PARAAS7xyU7meYxR77GXF6KJhRQ== 2023-10-30T12:16:43.524 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.1] 2023-10-30T12:16:43.524 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCSnj9lSMLjOhAAOk1G8FKKvOmsEVnb3edA4g== 2023-10-30T12:16:43.524 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.2] 2023-10-30T12:16:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCVnj9lGRwSORAA+MY93jndTp9U6easSNU8BQ== 2023-10-30T12:16:43.525 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.3] 2023-10-30T12:16:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCYnj9lU6lZNhAAS6wqyKIwtQuBsjCA0Myniw== 2023-10-30T12:16:43.525 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.x] 2023-10-30T12:16:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCMnj9lLVzCJxAAuT+90anVEc+rgZPakhM+rQ== 2023-10-30T12:16:43.526 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.4] 2023-10-30T12:16:43.526 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCfnj9lEKPuAhAAh+Vwi6VzNA4rXTG+dG6v7Q== 2023-10-30T12:16:43.526 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.5] 2023-10-30T12:16:43.526 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCinj9lgoSfABAA05kXJ3nFlB4MnavLDm/xWQ== 2023-10-30T12:16:43.526 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.6] 2023-10-30T12:16:43.527 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCknj9lGZO/ORAAq2fdkjKz73jutDA2mDzoew== 2023-10-30T12:16:43.527 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.7] 2023-10-30T12:16:43.527 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCnnj9lt+ugNxAAftrtI4ie0ednWjhT0Qv7rg== 2023-10-30T12:16:43.527 INFO:teuthology.orchestra.run.smithi148.stdout:[client.0] 2023-10-30T12:16:43.527 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCMnj9lK6p5KhAAKyfl1Ktv6Zcs6n0vQN3Qbw== 2023-10-30T12:16:43.527 INFO:teuthology.orchestra.run.smithi148.stdout:[client.1] 2023-10-30T12:16:43.528 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCMnj9lfhTwLRAAWAAlbLyM5Te6A7bnpP8RRw== 2023-10-30T12:16:43.546 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.y] 2023-10-30T12:16:43.546 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCMnj9lqD+aIhAAnyb0blX8YfsHIgb1Smgu8Q== 2023-10-30T12:16:43.547 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2023-10-30T12:16:43.547 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCQnj9l651PARAAS7xyU7meYxR77GXF6KJhRQ== 2023-10-30T12:16:43.547 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2023-10-30T12:16:43.547 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCSnj9lSMLjOhAAOk1G8FKKvOmsEVnb3edA4g== 2023-10-30T12:16:43.547 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2023-10-30T12:16:43.547 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCVnj9lGRwSORAA+MY93jndTp9U6easSNU8BQ== 2023-10-30T12:16:43.548 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.3] 2023-10-30T12:16:43.548 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCYnj9lU6lZNhAAS6wqyKIwtQuBsjCA0Myniw== 2023-10-30T12:16:43.548 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.x] 2023-10-30T12:16:43.548 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCMnj9lLVzCJxAAuT+90anVEc+rgZPakhM+rQ== 2023-10-30T12:16:43.548 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.4] 2023-10-30T12:16:43.549 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCfnj9lEKPuAhAAh+Vwi6VzNA4rXTG+dG6v7Q== 2023-10-30T12:16:43.549 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.5] 2023-10-30T12:16:43.549 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCinj9lgoSfABAA05kXJ3nFlB4MnavLDm/xWQ== 2023-10-30T12:16:43.549 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.6] 2023-10-30T12:16:43.549 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCknj9lGZO/ORAAq2fdkjKz73jutDA2mDzoew== 2023-10-30T12:16:43.550 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.7] 2023-10-30T12:16:43.550 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCnnj9lt+ugNxAAftrtI4ie0ednWjhT0Qv7rg== 2023-10-30T12:16:43.550 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2023-10-30T12:16:43.550 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCMnj9lK6p5KhAAKyfl1Ktv6Zcs6n0vQN3Qbw== 2023-10-30T12:16:43.550 INFO:teuthology.orchestra.run.smithi150.stdout:[client.1] 2023-10-30T12:16:43.551 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCMnj9lfhTwLRAAWAAlbLyM5Te6A7bnpP8RRw== 2023-10-30T12:16:43.552 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:16:43.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:16:43.646 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.649 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.702 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.704 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.756 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.759 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.811 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.866 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:16:43.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:16:43.921 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.975 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:43.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:44.028 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:44.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:44.082 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:44.084 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:16:44.136 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:16:44.139 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:16:44.194 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:16:44.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:16:44.252 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T12:16:44.252 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T12:16:44.280 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:16:44.376 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T12:16:44.403 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T12:16:44.470 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:16:44.568 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T12:16:44.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T12:16:44.623 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:16:44.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T12:16:44.747 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:44.749 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:16:44.801 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T12:16:44.802 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T12:16:44.802 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:16:44.803 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-10-30T12:16:44.805 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:16:44.805 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T12:16:44.805 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T12:16:44.806 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-10-30T12:16:44.808 INFO:tasks.ceph.mon.c:Started 2023-10-30T12:16:44.808 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T12:16:44.809 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:16:44.809 DEBUG:teuthology.orchestra.run.smithi150:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-10-30T12:16:44.843 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:16:44.844 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T12:16:44.844 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-30T12:16:44.845 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-10-30T12:16:44.847 INFO:tasks.ceph.mgr.y:Started 2023-10-30T12:16:44.847 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T12:16:44.848 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-10-30T12:16:44.849 INFO:tasks.ceph.mgr.x:Started 2023-10-30T12:16:44.850 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T12:16:44.850 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T12:16:44.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.950+0000 7fc887a94640 10 monclient: get_monmap_and_config 2023-10-30T12:16:44.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.950+0000 7fc887a94640 10 monclient: build_initial_monmap 2023-10-30T12:16:44.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.950+0000 7fc887a94640 10 monclient: monmap: 2023-10-30T12:16:44.955 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:16:44.955 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:16:44.955 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:44.951955+0000 2023-10-30T12:16:44.956 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:44.951955+0000 2023-10-30T12:16:44.956 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:44.956 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:16:44.956 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:16:44.957 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:16:44.957 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:16:44.957 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:16:44.957 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:16:44.957 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:16:44.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.951+0000 7fc887a94640 1 Processor -- start 2023-10-30T12:16:44.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.951+0000 7fc887a94640 1 -- start start 2023-10-30T12:16:44.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.951+0000 7fc887a94640 10 monclient: init 2023-10-30T12:16:44.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:44.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8800a8f60 addr v1:172.21.15.150:6789/0 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 0x7fc88013e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 10 monclient(hunting): picked mon.noname-b con 0x7fc8800a9340 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc880142780 addr v1:172.21.15.148:6789/0 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8800aa450 con 0x7fc880142780 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8800aa730 con 0x7fc8800a8f60 2023-10-30T12:16:44.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc880067110 con 0x7fc8800a9340 2023-10-30T12:16:44.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:44.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc887a94640 10 monclient(hunting): authenticate will time out at 1497.157214s 2023-10-30T12:16:44.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc88502f640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 msgr2=0x7fc88013e7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2023-10-30T12:16:44.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc886031640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc880142780 legacy=0x7fc88013ecf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2023-10-30T12:16:44.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc88502f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 0x7fc88013e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:44.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:44.952+0000 7fc885830640 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc8800a8f60 legacy=0x7fc880142070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2023-10-30T12:16:45.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:45.153+0000 7fc88502f640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 msgr2=0x7fc88013e7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2023-10-30T12:16:45.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:45.153+0000 7fc886031640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc880142780 legacy=0x7fc88013ecf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2023-10-30T12:16:45.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:45.153+0000 7fc88502f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 0x7fc88013e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:45.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:45.153+0000 7fc885830640 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc8800a8f60 legacy=0x7fc880142070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2023-10-30T12:16:45.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:45.554+0000 7fc88502f640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 msgr2=0x7fc88013e7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2023-10-30T12:16:45.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:45.554+0000 7fc886031640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc880142780 legacy=0x7fc88013ecf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2023-10-30T12:16:45.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:45.554+0000 7fc88502f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 0x7fc88013e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:45.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:45.554+0000 7fc885830640 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc8800a8f60 legacy=0x7fc880142070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2023-10-30T12:16:46.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:46.355+0000 7fc88502f640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 msgr2=0x7fc88013e7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2023-10-30T12:16:46.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:46.355+0000 7fc886031640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc880142780 legacy=0x7fc88013ecf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2023-10-30T12:16:46.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:46.355+0000 7fc88502f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 0x7fc88013e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:46.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:46.355+0000 7fc885830640 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc8800a8f60 legacy=0x7fc880142070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2023-10-30T12:16:47.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 10 monclient(hunting): tick 2023-10-30T12:16:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:16:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:16:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:16:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 msgr2=0x7fc88013e7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:47.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8800a9340 0x7fc88013e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc8800a8f60 legacy=0x7fc880142070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:47.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc880142780 legacy=0x7fc88013ecf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:47.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 10 monclient: _add_conns ranks=[4,3] 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 0x7fc864005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc8640036d0 addr v2:172.21.15.150:3300/0 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 0x7fc864008660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc8640061a0 addr v2:172.21.15.148:3300/0 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc880067110 con 0x7fc8640061a0 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc864008d10 con 0x7fc8640036d0 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc8777fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc88502f640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 msgr2=0x7fc864008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc88502f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 0x7fc864008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:47.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc885830640 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 msgr2=0x7fc864005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2023-10-30T12:16:47.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:47.952+0000 7fc885830640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 0x7fc864005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:48.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:48.153+0000 7fc88502f640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 msgr2=0x7fc864008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2023-10-30T12:16:48.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:48.153+0000 7fc88502f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 0x7fc864008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:48.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:48.153+0000 7fc885830640 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 msgr2=0x7fc864005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2023-10-30T12:16:48.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:48.153+0000 7fc885830640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 0x7fc864005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:48.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:48.554+0000 7fc88502f640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 msgr2=0x7fc864008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2023-10-30T12:16:48.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:48.554+0000 7fc88502f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 0x7fc864008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:48.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:48.554+0000 7fc885830640 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 msgr2=0x7fc864005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2023-10-30T12:16:48.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:48.554+0000 7fc885830640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 0x7fc864005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:49.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:49.355+0000 7fc88502f640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 msgr2=0x7fc864008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2023-10-30T12:16:49.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:49.355+0000 7fc88502f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 0x7fc864008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:49.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:49.355+0000 7fc885830640 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 msgr2=0x7fc864005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2023-10-30T12:16:49.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:49.355+0000 7fc885830640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 0x7fc864005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:50.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 10 monclient(hunting): tick 2023-10-30T12:16:50.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:16:50.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:16:50.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:16:50.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 msgr2=0x7fc864005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:50.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8640036d0 0x7fc864005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:50.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 msgr2=0x7fc864008660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:50.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc8640061a0 0x7fc864008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:50.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc886031640 1 -- reap_dead start 2023-10-30T12:16:50.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-10-30T12:16:50.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc864003b40 addr v1:172.21.15.150:6789/0 2023-10-30T12:16:50.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 0x7fc864008190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:50.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc864007d70 addr v2:172.21.15.150:3300/0 2023-10-30T12:16:50.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 0x7fc86400cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-b con 0x7fc8640086d0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:16:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8800aa450 con 0x7fc864003b40 2023-10-30T12:16:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc880067110 con 0x7fc864007d70 2023-10-30T12:16:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc864008d10 con 0x7fc8640086d0 2023-10-30T12:16:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc8777fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc886031640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 msgr2=0x7fc86400cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2023-10-30T12:16:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc885830640 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc864003b40 legacy=0x7fc864006650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2023-10-30T12:16:50.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc886031640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 0x7fc86400cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:50.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc88502f640 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 msgr2=0x7fc864008190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2023-10-30T12:16:50.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:50.952+0000 7fc88502f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 0x7fc864008190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:51.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:51.153+0000 7fc886031640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 msgr2=0x7fc86400cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2023-10-30T12:16:51.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:51.153+0000 7fc88502f640 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 msgr2=0x7fc864008190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2023-10-30T12:16:51.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:51.153+0000 7fc886031640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 0x7fc86400cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:51.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:51.153+0000 7fc885830640 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc864003b40 legacy=0x7fc864006650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2023-10-30T12:16:51.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:51.153+0000 7fc88502f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 0x7fc864008190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:51.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:51.554+0000 7fc886031640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 msgr2=0x7fc86400cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2023-10-30T12:16:51.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:51.554+0000 7fc885830640 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc864003b40 legacy=0x7fc864006650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2023-10-30T12:16:51.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:51.554+0000 7fc88502f640 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 msgr2=0x7fc864008190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2023-10-30T12:16:51.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:51.554+0000 7fc88502f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 0x7fc864008190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:51.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:51.554+0000 7fc886031640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 0x7fc86400cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:52.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:52.355+0000 7fc886031640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 msgr2=0x7fc86400cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2023-10-30T12:16:52.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:52.355+0000 7fc886031640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 0x7fc86400cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:52.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:52.355+0000 7fc885830640 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc864003b40 legacy=0x7fc864006650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2023-10-30T12:16:52.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:52.355+0000 7fc88502f640 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 msgr2=0x7fc864008190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2023-10-30T12:16:52.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:52.355+0000 7fc88502f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 0x7fc864008190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:52.494 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:00:07.542 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:52.807 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:00:07.854 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:52.835 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:00:07.908 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:52.835 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:00:07.908 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.158 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:00:08.232 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.159 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:00:08.232 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 10 monclient(hunting): tick 2023-10-30T12:16:53.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:16:53.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:16:53.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:16:53.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 msgr2=0x7fc86400cf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:53.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc8640086d0 0x7fc86400cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:53.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 msgr2=0x7fc864008190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:53.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc864007d70 0x7fc864008190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:53.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc864003b40 legacy=0x7fc864006650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:53.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 10 monclient: _add_conns ranks=[0,3] 2023-10-30T12:16:53.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc86400d670 addr v1:172.21.15.148:6789/0 2023-10-30T12:16:53.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:53.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc864011800 addr v2:172.21.15.148:3300/0 2023-10-30T12:16:53.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8800aa450 con 0x7fc86400d670 2023-10-30T12:16:53.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:53.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc880067110 con 0x7fc864011800 2023-10-30T12:16:53.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc8777fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:53.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc885830640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc86400d670 legacy=0x7fc86400da80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2023-10-30T12:16:53.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc88502f640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 msgr2=0x7fc864013bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2023-10-30T12:16:53.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:53.952+0000 7fc88502f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:54.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.153+0000 7fc88502f640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 msgr2=0x7fc864013bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2023-10-30T12:16:54.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.153+0000 7fc885830640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc86400d670 legacy=0x7fc86400da80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2023-10-30T12:16:54.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.153+0000 7fc88502f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:54.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.644+0000 7fc88502f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:54.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.693+0000 7fc88502f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44500/0 (socket says 172.21.15.148:44500) 2023-10-30T12:16:54.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.693+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 learned_addr learned my addr 172.21.15.148:0/2916255829 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:16:54.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.693+0000 7fc88502f640 10 monclient(hunting): get_auth_request con 0x7fc864011800 auth_method 0 2023-10-30T12:16:54.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.693+0000 7fc88502f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:54.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.693+0000 7fc88502f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:54.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.693+0000 7fc88502f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:54.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.834+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 msgr2=0x7fc864013bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:54.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.834+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 msgr2=0x7fc864013bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:54.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.834+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:54.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.834+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:54.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.848+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> v1:172.21.15.148:6789/0 conn(0x7fc86400d670 legacy=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:54.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.848+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> v1:172.21.15.148:6789/0 conn(0x7fc86400d670 legacy=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:54.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:54.848+0000 7fc885830640 1 --1- 172.21.15.148:0/2916255829 >> v1:172.21.15.148:6789/0 conn(0x7fc86400d670 0x7fc86400da80 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2023-10-30T12:16:55.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:55.688+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:55.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:55.689+0000 7fc88502f640 10 monclient(hunting): get_auth_request con 0x7fc864011800 auth_method 0 2023-10-30T12:16:55.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:55.689+0000 7fc88502f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:55.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:55.689+0000 7fc88502f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:55.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:55.689+0000 7fc88502f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:55.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:55.711+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 msgr2=0x7fc864013bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:55.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:55.711+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 msgr2=0x7fc864013bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:55.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:55.711+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:55.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:55.711+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:56.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.952+0000 7fc8777fe640 10 monclient(hunting): tick 2023-10-30T12:16:56.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.952+0000 7fc8777fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:16:56.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.952+0000 7fc8777fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:16:56.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.952+0000 7fc8777fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:16:56.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.952+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 msgr2=0x7fc864013bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:56.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.952+0000 7fc8777fe640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864011800 0x7fc864013bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:56.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.952+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 >> v1:172.21.15.148:6789/0 conn(0x7fc86400d670 legacy=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:56.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.952+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 reap_dead start 2023-10-30T12:16:56.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.952+0000 7fc8777fe640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-10-30T12:16:56.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.952+0000 7fc8777fe640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:56.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.953+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-b con 0x7fc864008190 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:16:56.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.953+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc864005b50 addr v1:172.21.15.150:6789/0 2023-10-30T12:16:56.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.953+0000 7fc8777fe640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:56.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.953+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc864010ee0 addr v2:172.21.15.150:3300/0 2023-10-30T12:16:56.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.953+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8800aa730 con 0x7fc864005b50 2023-10-30T12:16:56.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.953+0000 7fc8777fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:56.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.953+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc880067110 con 0x7fc864010ee0 2023-10-30T12:16:56.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.953+0000 7fc8777fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:56.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.953+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc864008d10 con 0x7fc864008190 2023-10-30T12:16:56.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.953+0000 7fc8777fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:56.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.978+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:56.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.979+0000 7fc885830640 10 monclient(hunting): get_auth_request con 0x7fc864008190 auth_method 0 2023-10-30T12:16:56.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.979+0000 7fc885830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:56.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.979+0000 7fc885830640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:56.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:56.979+0000 7fc885830640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:57.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.001+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:57.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.007+0000 7fc886031640 10 monclient(hunting): get_auth_request con 0x7fc864010ee0 auth_method 0 2023-10-30T12:16:57.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.007+0000 7fc886031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:57.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.007+0000 7fc886031640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:57.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.007+0000 7fc886031640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:57.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.008+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 msgr2=0x7fc864007600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:16:57.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.008+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 msgr2=0x7fc864007600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:57.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.008+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:57.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.008+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:57.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.015+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 msgr2=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:57.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.015+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 msgr2=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:57.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.015+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:57.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.015+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:57.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.209+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:57.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.209+0000 7fc886031640 10 monclient(hunting): get_auth_request con 0x7fc864010ee0 auth_method 0 2023-10-30T12:16:57.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.209+0000 7fc886031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:57.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.209+0000 7fc886031640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:57.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.209+0000 7fc886031640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:57.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.210+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 msgr2=0x7fc864007600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:57.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.210+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 msgr2=0x7fc864007600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:57.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.210+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:57.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.210+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:57.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.216+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:57.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.216+0000 7fc885830640 10 monclient(hunting): get_auth_request con 0x7fc864008190 auth_method 0 2023-10-30T12:16:57.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.216+0000 7fc885830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:57.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.216+0000 7fc885830640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:57.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.216+0000 7fc885830640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:57.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.217+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 msgr2=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:57.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.217+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 msgr2=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:57.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.217+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:57.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.217+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:57.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.611+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:57.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.612+0000 7fc886031640 10 monclient(hunting): get_auth_request con 0x7fc864010ee0 auth_method 0 2023-10-30T12:16:57.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.612+0000 7fc886031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:57.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.612+0000 7fc886031640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:57.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.612+0000 7fc886031640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:57.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.613+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 msgr2=0x7fc864007600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:57.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.613+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 msgr2=0x7fc864007600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:57.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.613+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:57.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.613+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:57.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.618+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:57.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.618+0000 7fc885830640 10 monclient(hunting): get_auth_request con 0x7fc864008190 auth_method 0 2023-10-30T12:16:57.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.618+0000 7fc885830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:57.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.618+0000 7fc885830640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:57.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.618+0000 7fc885830640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:57.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.619+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 msgr2=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:57.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.619+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 msgr2=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:57.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.619+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:57.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:57.619+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:58.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.415+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:58.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.415+0000 7fc886031640 10 monclient(hunting): get_auth_request con 0x7fc864010ee0 auth_method 0 2023-10-30T12:16:58.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.415+0000 7fc886031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:58.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.415+0000 7fc886031640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:58.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.415+0000 7fc886031640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:58.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.416+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 msgr2=0x7fc864007600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:58.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.416+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 msgr2=0x7fc864007600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:58.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.416+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:58.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.416+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:58.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.421+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:58.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.421+0000 7fc885830640 10 monclient(hunting): get_auth_request con 0x7fc864008190 auth_method 0 2023-10-30T12:16:58.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.421+0000 7fc885830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:58.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.421+0000 7fc885830640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:58.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.421+0000 7fc885830640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:58.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.422+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 msgr2=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:58.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.422+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 msgr2=0x7fc86400da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:58.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.422+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:58.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:58.422+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:59.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 10 monclient(hunting): tick 2023-10-30T12:16:59.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:16:59.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:16:59.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:16:59.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 msgr2=0x7fc86400da80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:59.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc864008190 0x7fc86400da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:59.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 msgr2=0x7fc864007600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:59.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc864010ee0 0x7fc864007600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:59.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 >> v1:172.21.15.150:6789/0 conn(0x7fc864005b50 legacy=0x7fc8640107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:59.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 10 monclient: _add_conns ranks=[0,3] 2023-10-30T12:16:59.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc864011360 addr v1:172.21.15.148:6789/0 2023-10-30T12:16:59.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:59.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc86400cf70 addr v2:172.21.15.148:3300/0 2023-10-30T12:16:59.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc864015310 con 0x7fc864011360 2023-10-30T12:16:59.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:59.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc880067110 con 0x7fc86400cf70 2023-10-30T12:16:59.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.953+0000 7fc8777fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:59.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.954+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:59.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.954+0000 7fc88502f640 10 monclient(hunting): get_auth_request con 0x7fc86400cf70 auth_method 0 2023-10-30T12:16:59.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.954+0000 7fc88502f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:59.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.954+0000 7fc88502f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:59.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.954+0000 7fc88502f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:59.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.955+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 msgr2=0x7fc864007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:59.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.955+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 msgr2=0x7fc864007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:59.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.955+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:59.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:16:59.955+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:00.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.156+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:00.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.157+0000 7fc88502f640 10 monclient(hunting): get_auth_request con 0x7fc86400cf70 auth_method 0 2023-10-30T12:17:00.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.157+0000 7fc88502f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:00.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.157+0000 7fc88502f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:00.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.157+0000 7fc88502f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:00.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.157+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 msgr2=0x7fc864007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:00.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.157+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 msgr2=0x7fc864007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:00.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.157+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:00.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.157+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:00.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.558+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:00.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.559+0000 7fc88502f640 10 monclient(hunting): get_auth_request con 0x7fc86400cf70 auth_method 0 2023-10-30T12:17:00.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.559+0000 7fc88502f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:00.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.559+0000 7fc88502f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:00.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.559+0000 7fc88502f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:00.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.559+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 msgr2=0x7fc864007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:00.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.559+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 msgr2=0x7fc864007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:00.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.559+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:00.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:00.559+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:01.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:01.361+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:01.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:01.361+0000 7fc88502f640 10 monclient(hunting): get_auth_request con 0x7fc86400cf70 auth_method 0 2023-10-30T12:17:01.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:01.361+0000 7fc88502f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:01.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:01.361+0000 7fc88502f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:01.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:01.361+0000 7fc88502f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:01.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:01.362+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 msgr2=0x7fc864007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:01.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:01.362+0000 7fc88502f640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 msgr2=0x7fc864007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:01.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:01.362+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:01.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:01.362+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:17:02.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 10 monclient(hunting): tick 2023-10-30T12:17:02.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:17:02.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:17:02.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:17:02.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 msgr2=0x7fc864007b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:02.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc86400cf70 0x7fc864007b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:02.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 >> v1:172.21.15.148:6789/0 conn(0x7fc864011360 legacy=0x7fc864011be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:02.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-10-30T12:17:02.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:02.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 reap_dead start 2023-10-30T12:17:02.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc864015520 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:02.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc864008190 addr v1:172.21.15.148:6789/0 2023-10-30T12:17:02.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:02.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc86400da80 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:02.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc864005b50 con 0x7fc864008190 2023-10-30T12:17:02.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:02.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc880067110 con 0x7fc864015520 2023-10-30T12:17:02.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:02.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 1 -- 172.21.15.148:0/2916255829 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc864008d10 con 0x7fc86400da80 2023-10-30T12:17:02.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc8777fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:02.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.954+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:02.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc886031640 10 monclient(hunting): get_auth_request con 0x7fc86400da80 auth_method 0 2023-10-30T12:17:02.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc886031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:02.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc886031640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:02.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc886031640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:02.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:02.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc885830640 10 monclient(hunting): get_auth_request con 0x7fc864015520 auth_method 0 2023-10-30T12:17:02.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc885830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:02.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc885830640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:02.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc885830640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:02.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 msgr2=0x7fc864015ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:17:02.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 msgr2=0x7fc864015ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:02.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.955+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:02.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.956+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:02.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.956+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 msgr2=0x7fc864015900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:02.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.956+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 msgr2=0x7fc864015900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:02.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.956+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:02.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:02.956+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:03.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.158+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:03.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.159+0000 7fc886031640 10 monclient(hunting): get_auth_request con 0x7fc86400da80 auth_method 0 2023-10-30T12:17:03.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.159+0000 7fc886031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:03.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.159+0000 7fc886031640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:03.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.159+0000 7fc886031640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:03.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.159+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 msgr2=0x7fc864015ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:03.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.159+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 msgr2=0x7fc864015ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:03.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.159+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:03.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.160+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:03.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.160+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:03.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.167+0000 7fc885830640 10 monclient(hunting): get_auth_request con 0x7fc864015520 auth_method 0 2023-10-30T12:17:03.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.167+0000 7fc885830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:03.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.167+0000 7fc885830640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:03.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.167+0000 7fc885830640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:03.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.169+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 msgr2=0x7fc864015900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:17:03.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.169+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 msgr2=0x7fc864015900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:03.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.169+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:03.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.169+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:03.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.561+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:03.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.562+0000 7fc886031640 10 monclient(hunting): get_auth_request con 0x7fc86400da80 auth_method 0 2023-10-30T12:17:03.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.562+0000 7fc886031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:03.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.562+0000 7fc886031640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:03.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.562+0000 7fc886031640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:03.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.562+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 msgr2=0x7fc864015ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:03.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.562+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 msgr2=0x7fc864015ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:03.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.562+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:03.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.562+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:03.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.597+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:03.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.614+0000 7fc885830640 10 monclient(hunting): get_auth_request con 0x7fc864015520 auth_method 0 2023-10-30T12:17:03.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.614+0000 7fc885830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:03.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.614+0000 7fc885830640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:03.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.614+0000 7fc885830640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:03.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.639+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 msgr2=0x7fc864015900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.639+0000 7fc885830640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 msgr2=0x7fc864015900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.639+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:03.639+0000 7fc885830640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:04.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.201+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991540729 0 0) 0x7fc870008450 con 0x7fc864008190 2023-10-30T12:17:04.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.201+0000 7fc88482e640 10 monclient(hunting): _negotiate 2023-10-30T12:17:04.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.201+0000 7fc88482e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.201+0000 7fc88482e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.201+0000 7fc88482e640 10 monclient(hunting): my global_id is 4098 2023-10-30T12:17:04.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.201+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c003520 con 0x7fc864008190 2023-10-30T12:17:04.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.259+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 300782078 0 0) 0x7fc870008730 con 0x7fc864008190 2023-10-30T12:17:04.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.259+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 msgr2=0x7fc864015ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:04.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.259+0000 7fc88482e640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.150:3300/0 conn(0x7fc86400da80 0x7fc864015ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 msgr2=0x7fc864015900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:04.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 --2- 172.21.15.148:0/2916255829 >> v2:172.21.15.148:3300/0 conn(0x7fc864015520 0x7fc864015900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient: _finish_hunting 0 2023-10-30T12:17:04.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:17:04.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:04.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.148:6789/0 2023-10-30T12:17:04.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc880141b70 con 0x7fc864008190 2023-10-30T12:17:04.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient: _finish_auth 0 2023-10-30T12:17:04.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient: _check_auth_tickets 2023-10-30T12:17:04.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:04.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1567732308 0 0) 0x7fc87000f040 con 0x7fc864008190 2023-10-30T12:17:04.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:17:04.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient: dump: 2023-10-30T12:17:04.266 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:04.266 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:04.267 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:04.267 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:04.267 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.267 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:04.268 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:04.268 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:04.268 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:04.268 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:04.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] but i'm connected to v1:172.21.15.148:6789/0, reconnecting 2023-10-30T12:17:04.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:04.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 >> v1:172.21.15.148:6789/0 conn(0x7fc864008190 legacy=0x7fc8640065f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:17:04.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc86c003970 0x7fc86c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient(hunting): picked mon.c con 0x7fc86c003970 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:04.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc86c0047c0 0x7fc86c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient(hunting): picked mon.a con 0x7fc86c0047c0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:04.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c005180 0x7fc86c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient(hunting): picked mon.b con 0x7fc86c005180 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:04.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc880067110 con 0x7fc86c0047c0 2023-10-30T12:17:04.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc864008d10 con 0x7fc86c005180 2023-10-30T12:17:04.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c00c0d0 con 0x7fc86c003970 2023-10-30T12:17:04.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:04.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.260+0000 7fc88482e640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:17:04.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.263+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c005180 0x7fc86c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.264+0000 7fc886031640 10 monclient(hunting): get_auth_request con 0x7fc86c005180 auth_method 0 2023-10-30T12:17:04.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.264+0000 7fc886031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.264+0000 7fc886031640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.264+0000 7fc886031640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:04.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.264+0000 7fc886031640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.264+0000 7fc886031640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.264+0000 7fc886031640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:04.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.301+0000 7fc88502f640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc86c0047c0 0x7fc86c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2023-10-30T12:17:04.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc86c003970 msgr2=0x7fc86c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:04.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc86c003970 0x7fc86c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc86c0047c0 msgr2=0x7fc86c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:04.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc86c0047c0 0x7fc86c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 10 monclient: _finish_hunting 0 2023-10-30T12:17:04.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 1 monclient: found mon.b 2023-10-30T12:17:04.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:04.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:04.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc870008730 con 0x7fc86c005180 2023-10-30T12:17:04.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 10 monclient: _finish_auth 0 2023-10-30T12:17:04.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 10 monclient: _check_auth_tickets 2023-10-30T12:17:04.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc886031640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:04.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc887a94640 5 monclient: authenticate success, global_id 4098 2023-10-30T12:17:04.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.315+0000 7fc887a94640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:04.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.316+0000 7fc886031640 1 -- 172.21.15.148:0/2916255829 reap_dead start 2023-10-30T12:17:04.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.326+0000 7fc886031640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c005180 0x7fc86c00b980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc8800720b0 tx=0x7fc86c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb8a632086bf23b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc878031070 con 0x7fc86c005180 2023-10-30T12:17:04.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc88482e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc88482e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:04.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc88482e640 10 monclient: dump: 2023-10-30T12:17:04.385 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:04.385 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:04.385 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:04.385 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:04.386 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.386 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:04.386 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:04.386 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:04.387 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:04.387 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:04.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87802da30 con 0x7fc86c005180 2023-10-30T12:17:04.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc88482e640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:04.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc88482e640 1 -- 172.21.15.148:0/2916255829 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87802ddf0 con 0x7fc86c005180 2023-10-30T12:17:04.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc88482e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc88482e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:04.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc88482e640 10 monclient: dump: 2023-10-30T12:17:04.388 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:04.389 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:04.389 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:04.389 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:04.389 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.390 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:04.390 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:04.390 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:04.390 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:04.391 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:04.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc887a94640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:04.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.382+0000 7fc887a94640 4 set_mon_vals no callback set 2023-10-30T12:17:04.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.383+0000 7fc887a94640 10 monclient: shutdown 2023-10-30T12:17:04.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.383+0000 7fc887a94640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:04.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.383+0000 7fc887a94640 1 -- 172.21.15.148:0/2916255829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c005180 msgr2=0x7fc86c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.383+0000 7fc887a94640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c005180 0x7fc86c00b980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc8800720b0 tx=0x7fc86c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.383+0000 7fc887a94640 1 -- 172.21.15.148:0/2916255829 shutdown_connections 2023-10-30T12:17:04.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.383+0000 7fc887a94640 1 --2- 172.21.15.148:0/2916255829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c005180 0x7fc86c00b980 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.383+0000 7fc887a94640 1 -- 172.21.15.148:0/2916255829 >> 172.21.15.148:0/2916255829 conn(0x7fc8800b74e0 msgr2=0x7fc8800b78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:04.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.383+0000 7fc887a94640 1 -- 172.21.15.148:0/2916255829 shutdown_connections 2023-10-30T12:17:04.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.383+0000 7fc887a94640 1 -- 172.21.15.148:0/2916255829 wait complete. 2023-10-30T12:17:04.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.384+0000 7fc887a94640 10 monclient: build_initial_monmap 2023-10-30T12:17:04.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.384+0000 7fc887a94640 10 monclient: monmap: 2023-10-30T12:17:04.397 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:17:04.397 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:04.397 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:04.397 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:17:04.398 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:04.398 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:04.398 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:04.398 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:04.399 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:04.399 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:04.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.384+0000 7fc887a94640 1 librados: starting msgr at 2023-10-30T12:17:04.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.384+0000 7fc887a94640 1 librados: starting objecter 2023-10-30T12:17:04.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.384+0000 7fc887a94640 1 Processor -- start 2023-10-30T12:17:04.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.384+0000 7fc887a94640 1 -- start start 2023-10-30T12:17:04.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.384+0000 7fc887a94640 1 librados: setting wanted keys 2023-10-30T12:17:04.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.384+0000 7fc887a94640 1 librados: calling monclient init 2023-10-30T12:17:04.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.384+0000 7fc887a94640 10 monclient: init 2023-10-30T12:17:04.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:04.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:17:04.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc86c005180 0x7fc88013a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 10 monclient(hunting): picked mon.noname-c con 0x7fc86c005180 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:04.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8800b0470 0x7fc880142ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 10 monclient(hunting): picked mon.noname-a con 0x7fc8800b0470 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:04.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8801d6c20 0x7fc8801d97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 10 monclient(hunting): picked mon.noname-b con 0x7fc8801d6c20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:04.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88013c1f0 con 0x7fc8800b0470 2023-10-30T12:17:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8800abb70 con 0x7fc8801d6c20 2023-10-30T12:17:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8801d6700 con 0x7fc86c005180 2023-10-30T12:17:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc887a94640 10 monclient(hunting): authenticate will time out at 1516.590421s 2023-10-30T12:17:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.385+0000 7fc885830640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc86c005180 0x7fc88013a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.386+0000 7fc885830640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc86c005180 0x7fc88013a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:52994/0 (socket says 172.21.15.148:52994) 2023-10-30T12:17:04.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.386+0000 7fc885830640 1 -- 172.21.15.148:0/16906622 learned_addr learned my addr 172.21.15.148:0/16906622 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:04.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.386+0000 7fc885830640 10 monclient(hunting): get_auth_request con 0x7fc86c005180 auth_method 0 2023-10-30T12:17:04.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.386+0000 7fc885830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.386+0000 7fc885830640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.386+0000 7fc885830640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.386+0000 7fc885830640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.386+0000 7fc885830640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.386+0000 7fc885830640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.389+0000 7fc885830640 10 monclient(hunting): handle_auth_done global_id 4106 payload 824 2023-10-30T12:17:04.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.389+0000 7fc885830640 1 -- 172.21.15.148:0/16906622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8801d6c20 msgr2=0x7fc8801d97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.389+0000 7fc885830640 1 --2- 172.21.15.148:0/16906622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8801d6c20 0x7fc8801d97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.389+0000 7fc885830640 1 -- 172.21.15.148:0/16906622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8800b0470 msgr2=0x7fc880142ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.389+0000 7fc885830640 1 --2- 172.21.15.148:0/16906622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8800b0470 0x7fc880142ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.389+0000 7fc885830640 10 monclient: _finish_hunting 0 2023-10-30T12:17:04.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.389+0000 7fc885830640 1 monclient: found mon.noname-c 2023-10-30T12:17:04.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.389+0000 7fc885830640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:04.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.389+0000 7fc885830640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:17:04.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.389+0000 7fc885830640 1 -- 172.21.15.148:0/16906622 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc880143410 con 0x7fc86c005180 2023-10-30T12:17:04.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.392+0000 7fc885830640 1 --2- 172.21.15.148:0/16906622 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc86c005180 0x7fc88013a620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc86400e230 tx=0x7fc87c00cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.392+0000 7fc8767fc640 10 client.?.objecter ms_handle_connect 0x7fc86c005180 2023-10-30T12:17:04.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.392+0000 7fc8767fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:04.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 1 -- 172.21.15.148:0/16906622 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c012040 con 0x7fc86c005180 2023-10-30T12:17:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 10 monclient: dump: 2023-10-30T12:17:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:04.463 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 10 monclient: _finish_auth 0 2023-10-30T12:17:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 10 monclient: _check_auth_tickets 2023-10-30T12:17:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 1 -- 172.21.15.148:0/16906622 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c018450 con 0x7fc86c005180 2023-10-30T12:17:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 1 -- 172.21.15.148:0/16906622 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c018810 con 0x7fc86c005180 2023-10-30T12:17:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc887a94640 5 monclient: authenticate success, global_id 4106 2023-10-30T12:17:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc887a94640 10 monclient: _renew_subs 2023-10-30T12:17:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc887a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc887a94640 1 -- 172.21.15.148:0/16906622 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8801d70d0 con 0x7fc86c005180 2023-10-30T12:17:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc877fff640 4 set_mon_vals no callback set 2023-10-30T12:17:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:04.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.457+0000 7fc8767fc640 10 monclient: dump: 2023-10-30T12:17:04.467 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:04.467 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:04.467 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:04.469 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.458+0000 7fc887a94640 10 client.4106.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:04.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.458+0000 7fc887a94640 10 monclient: _renew_subs 2023-10-30T12:17:04.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.458+0000 7fc887a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:04.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.458+0000 7fc887a94640 1 -- 172.21.15.148:0/16906622 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8801d6490 con 0x7fc86c005180 2023-10-30T12:17:04.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.458+0000 7fc887a94640 1 librados: init done 2023-10-30T12:17:04.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.459+0000 7fc887a94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:04.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.459+0000 7fc887a94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:04.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.459+0000 7fc887a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:04.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.459+0000 7fc887a94640 1 -- 172.21.15.148:0/16906622 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc880075550 con 0x7fc86c005180 2023-10-30T12:17:04.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.686+0000 7fc8767fc640 1 -- 172.21.15.148:0/16906622 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc87c018980 con 0x7fc86c005180 2023-10-30T12:17:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.686+0000 7fc8767fc640 1 -- 172.21.15.148:0/16906622 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc87c022a20 con 0x7fc86c005180 2023-10-30T12:17:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.686+0000 7fc8767fc640 10 client.4106.objecter ms_dispatch 0x7fc880000b90 osd_map(1..1 src has 1..1) v4 2023-10-30T12:17:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.686+0000 7fc8767fc640 3 client.4106.objecter handle_osd_map got epochs [1,1] > 0 2023-10-30T12:17:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.686+0000 7fc8767fc640 3 client.4106.objecter handle_osd_map decoding full epoch 1 2023-10-30T12:17:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.687+0000 7fc8767fc640 20 client.4106.objecter dump_active .. 0 homeless 2023-10-30T12:17:04.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.687+0000 7fc8767fc640 1 -- 172.21.15.148:0/16906622 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc87c01d070 con 0x7fc86c005180 2023-10-30T12:17:04.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.687+0000 7fc8767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:04.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.687+0000 7fc8767fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:04.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.757+0000 7fc887a94640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T12:17:04.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.757+0000 7fc887a94640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T12:17:04.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.757+0000 7fc887a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:04.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:04.757+0000 7fc887a94640 1 -- 172.21.15.148:0/16906622 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc8800678b0 con 0x7fc86c005180 2023-10-30T12:17:05.014 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:05.013+0000 7f4e91df1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T12:17:05.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.017+0000 7fc8767fc640 1 -- 172.21.15.148:0/16906622 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc87c018dc0 con 0x7fc86c005180 2023-10-30T12:17:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.017+0000 7fc8767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T12:17:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.017+0000 7fc8767fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-10-30T12:17:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:adjusted tunables profile to default 2023-10-30T12:17:05.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 10 librados: watch_flush enter 2023-10-30T12:17:05.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 10 librados: watch_flush exit 2023-10-30T12:17:05.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 20 client.4106.objecter shutdown clearing up homeless session... 2023-10-30T12:17:05.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 10 client.4106.objecter successfully canceled tick 2023-10-30T12:17:05.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 10 monclient: shutdown 2023-10-30T12:17:05.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:05.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 1 -- 172.21.15.148:0/16906622 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc86c005180 msgr2=0x7fc88013a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:05.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 1 --2- 172.21.15.148:0/16906622 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc86c005180 0x7fc88013a620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc86400e230 tx=0x7fc87c00cf40 comp rx=0 tx=0).stop 2023-10-30T12:17:05.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 1 -- 172.21.15.148:0/16906622 shutdown_connections 2023-10-30T12:17:05.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 1 --2- 172.21.15.148:0/16906622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8801d6c20 0x7fc8801d97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 1 --2- 172.21.15.148:0/16906622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8800b0470 0x7fc880142ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 1 --2- 172.21.15.148:0/16906622 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc86c005180 0x7fc88013a620 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 1 -- 172.21.15.148:0/16906622 >> 172.21.15.148:0/16906622 conn(0x7fc8800b74e0 msgr2=0x7fc8800b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:05.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 1 -- 172.21.15.148:0/16906622 shutdown_connections 2023-10-30T12:17:05.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 1 -- 172.21.15.148:0/16906622 wait complete. 2023-10-30T12:17:05.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:05.018+0000 7fc887a94640 1 librados: shutdown 2023-10-30T12:17:05.032 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T12:17:05.033 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T12:17:05.034 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:17:05.034 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T12:17:05.061 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:17:05.062 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T12:17:05.076 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:05.075+0000 7f4e91df1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T12:17:05.103 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:17:05.103 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T12:17:05.173 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:17:05.174 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T12:17:05.240 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:17:05.240 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T12:17:05.268 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:17:05.268 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T12:17:05.334 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:17:05.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-30T12:17:05.401 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:17:05.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-30T12:17:05.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 183f5799-ff32-4ada-b4ff-fd8fcbd82d08 0 2023-10-30T12:17:05.605 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.604+0000 7efe41f9e640 10 monclient: get_monmap_and_config 2023-10-30T12:17:05.606 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.604+0000 7efe41f9e640 10 monclient: build_initial_monmap 2023-10-30T12:17:05.606 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.604+0000 7efe41f9e640 10 monclient: monmap: 2023-10-30T12:17:05.606 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:05.606 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:17:05.605308+0000 2023-10-30T12:17:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:17:05.605308+0000 2023-10-30T12:17:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:05.608 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:05.608 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:05.608 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:05.608 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:05.608 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:05.611 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 1 Processor -- start 2023-10-30T12:17:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 1 -- start start 2023-10-30T12:17:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 10 monclient: init 2023-10-30T12:17:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-10-30T12:17:05.613 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0a8f80 0x7efe3c0b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:05.613 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 10 monclient(hunting): picked mon.noname-b con 0x7efe3c0a8f80 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:05.613 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efe3c0a9360 0x7efe3c1420d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 10 monclient(hunting): picked mon.noname-a con 0x7efe3c0a9360 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe3c142610 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3c141910 con 0x7efe3c142610 2023-10-30T12:17:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe3c066f80 con 0x7efe3c0a9360 2023-10-30T12:17:05.615 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:05.615 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c0a8a30 con 0x7efe3c0a8f80 2023-10-30T12:17:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.605+0000 7efe41f9e640 10 monclient(hunting): authenticate will time out at 1521.836768s 2023-10-30T12:17:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.606+0000 7efe4179d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7efe3c142610 0x7efe3c13ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47180/0 (socket says 172.21.15.150:47180) 2023-10-30T12:17:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.606+0000 7efe4179d640 1 -- 172.21.15.150:0/3772289187 learned_addr learned my addr 172.21.15.150:0/3772289187 (peer_addr_for_me v1:172.21.15.150:0/0) 2023-10-30T12:17:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.606+0000 7efe40f9c640 1 --2- 172.21.15.150:0/3772289187 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0a8f80 0x7efe3c0b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:05.617 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.606+0000 7efe3bfff640 1 --2- 172.21.15.150:0/3772289187 >> v2:172.21.15.148:3300/0 conn(0x7efe3c0a9360 0x7efe3c1420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:05.617 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.606+0000 7efe40f9c640 10 monclient(hunting): get_auth_request con 0x7efe3c0a8f80 auth_method 0 2023-10-30T12:17:05.617 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.606+0000 7efe40f9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:05.617 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.606+0000 7efe40f9c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:05.617 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.606+0000 7efe40f9c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:05.618 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.607+0000 7efe3bfff640 10 monclient(hunting): get_auth_request con 0x7efe3c0a9360 auth_method 0 2023-10-30T12:17:05.618 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.607+0000 7efe3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:05.618 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.607+0000 7efe3bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:05.618 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.607+0000 7efe3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:05.619 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.607+0000 7efe40f9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:05.619 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.607+0000 7efe40f9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:05.619 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.607+0000 7efe40f9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:05.619 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.607+0000 7efe3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:05.619 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.607+0000 7efe3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:05.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.607+0000 7efe3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:05.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.609+0000 7efe40f9c640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2023-10-30T12:17:05.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.609+0000 7efe40f9c640 1 -- 172.21.15.150:0/3772289187 >> v2:172.21.15.148:3300/0 conn(0x7efe3c0a9360 msgr2=0x7efe3c1420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:05.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.609+0000 7efe40f9c640 1 --2- 172.21.15.150:0/3772289187 >> v2:172.21.15.148:3300/0 conn(0x7efe3c0a9360 0x7efe3c1420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.609+0000 7efe40f9c640 1 -- 172.21.15.150:0/3772289187 >> v1:172.21.15.150:6789/0 conn(0x7efe3c142610 legacy=0x7efe3c13ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:05.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.609+0000 7efe40f9c640 10 monclient: _finish_hunting 0 2023-10-30T12:17:05.621 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.609+0000 7efe40f9c640 1 monclient: found mon.noname-b 2023-10-30T12:17:05.621 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.609+0000 7efe40f9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:05.621 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.609+0000 7efe40f9c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:17:05.621 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.609+0000 7efe40f9c640 1 -- 172.21.15.150:0/3772289187 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c141bb0 con 0x7efe3c0a8f80 2023-10-30T12:17:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.611+0000 7efe40f9c640 1 --2- 172.21.15.150:0/3772289187 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0a8f80 0x7efe3c0b1b00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efe240097a0 tx=0x7efe2402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d050dbe7980e12c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 1 -- 172.21.15.150:0/3772289187 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe240315a0 con 0x7efe3c0a8f80 2023-10-30T12:17:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:05.627 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 10 monclient: dump: 2023-10-30T12:17:05.627 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:05.627 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:05.627 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:05.627 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:05.628 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:05.628 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:05.628 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:05.628 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:05.628 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:05.629 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:05.629 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:05.629 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:05.629 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:05.629 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 1 -- 172.21.15.150:0/3772289187 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe24031710 con 0x7efe3c0a8f80 2023-10-30T12:17:05.630 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 5 monclient: authenticate success, global_id 4109 2023-10-30T12:17:05.630 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:05.630 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:05.630 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:05.630 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 1 -- 172.21.15.150:0/3772289187 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe24031ab0 con 0x7efe3c0a8f80 2023-10-30T12:17:05.631 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:05.631 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 4 set_mon_vals no callback set 2023-10-30T12:17:05.631 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:05.631 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe3b7fe640 10 monclient: dump: 2023-10-30T12:17:05.631 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:05.632 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:05.632 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:05.632 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:05.632 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:05.632 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:05.633 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:05.633 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:05.633 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:05.633 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:05.633 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 10 monclient: shutdown 2023-10-30T12:17:05.633 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:05.634 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 1 -- 172.21.15.150:0/3772289187 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0a8f80 msgr2=0x7efe3c0b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:05.634 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 1 --2- 172.21.15.150:0/3772289187 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0a8f80 0x7efe3c0b1b00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efe240097a0 tx=0x7efe2402b840 comp rx=0 tx=0).stop 2023-10-30T12:17:05.634 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 1 -- 172.21.15.150:0/3772289187 shutdown_connections 2023-10-30T12:17:05.634 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 1 --2- 172.21.15.150:0/3772289187 >> v2:172.21.15.148:3300/0 conn(0x7efe3c0a9360 0x7efe3c1420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.634 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 1 --2- 172.21.15.150:0/3772289187 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0a8f80 0x7efe3c0b1b00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.635 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 1 -- 172.21.15.150:0/3772289187 >> 172.21.15.150:0/3772289187 conn(0x7efe3c0b7180 msgr2=0x7efe3c0b7590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:05.635 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 1 -- 172.21.15.150:0/3772289187 shutdown_connections 2023-10-30T12:17:05.635 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.615+0000 7efe41f9e640 1 -- 172.21.15.150:0/3772289187 wait complete. 2023-10-30T12:17:05.635 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.616+0000 7efe41f9e640 10 monclient: build_initial_monmap 2023-10-30T12:17:05.635 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.616+0000 7efe41f9e640 10 monclient: monmap: 2023-10-30T12:17:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2023-10-30T12:17:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2023-10-30T12:17:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:05.637 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2023-10-30T12:17:05.637 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:05.637 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:05.637 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:05.637 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:05.637 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.616+0000 7efe41f9e640 1 librados: starting msgr at 2023-10-30T12:17:05.638 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.616+0000 7efe41f9e640 1 librados: starting objecter 2023-10-30T12:17:05.638 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.616+0000 7efe41f9e640 1 Processor -- start 2023-10-30T12:17:05.638 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 1 -- start start 2023-10-30T12:17:05.638 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 1 librados: setting wanted keys 2023-10-30T12:17:05.638 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 1 librados: calling monclient init 2023-10-30T12:17:05.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient: init 2023-10-30T12:17:05.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:05.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:17:05.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3c0a8f80 0x7efe3c141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:05.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient(hunting): picked mon.noname-a con 0x7efe3c0a8f80 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:05.640 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c0a9360 0x7efe3c1d6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:05.640 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient(hunting): picked mon.noname-b con 0x7efe3c0a9360 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:05.640 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0b0700 0x7efe3c1d9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:05.640 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient(hunting): picked mon.noname-c con 0x7efe3c0b0700 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:05.640 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c0b3090 con 0x7efe3c0a8f80 2023-10-30T12:17:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c1d6b10 con 0x7efe3c0a9360 2023-10-30T12:17:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c1d74e0 con 0x7efe3c0b0700 2023-10-30T12:17:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.617+0000 7efe41f9e640 10 monclient(hunting): authenticate will time out at 1521.848689s 2023-10-30T12:17:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c0a9360 0x7efe3c1d6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe4179d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0b0700 0x7efe3c1d9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe40f9c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3c0a8f80 0x7efe3c141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe4179d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0b0700 0x7efe3c1d9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.150:34556/0 (socket says 172.21.15.150:34556) 2023-10-30T12:17:05.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c0a9360 0x7efe3c1d6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53462/0 (socket says 172.21.15.150:53462) 2023-10-30T12:17:05.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe4179d640 1 -- 172.21.15.150:0/1692711410 learned_addr learned my addr 172.21.15.150:0/1692711410 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:05.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe4179d640 10 monclient(hunting): get_auth_request con 0x7efe3c0b0700 auth_method 0 2023-10-30T12:17:05.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe4179d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:05.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe4179d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe4179d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe3bfff640 10 monclient(hunting): get_auth_request con 0x7efe3c0a9360 auth_method 0 2023-10-30T12:17:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe3bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe40f9c640 10 monclient(hunting): get_auth_request con 0x7efe3c0a8f80 auth_method 0 2023-10-30T12:17:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe40f9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe40f9c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.618+0000 7efe40f9c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.619+0000 7efe4179d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.619+0000 7efe4179d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.619+0000 7efe4179d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.619+0000 7efe3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.619+0000 7efe3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.619+0000 7efe3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.619+0000 7efe40f9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.619+0000 7efe40f9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.619+0000 7efe40f9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.621+0000 7efe4179d640 10 monclient(hunting): handle_auth_done global_id 4112 payload 824 2023-10-30T12:17:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.621+0000 7efe4179d640 1 -- 172.21.15.150:0/1692711410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c0a9360 msgr2=0x7efe3c1d6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.621+0000 7efe4179d640 1 --2- 172.21.15.150:0/1692711410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c0a9360 0x7efe3c1d6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.648 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.621+0000 7efe4179d640 1 -- 172.21.15.150:0/1692711410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3c0a8f80 msgr2=0x7efe3c141160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:05.648 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.621+0000 7efe4179d640 1 --2- 172.21.15.150:0/1692711410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3c0a8f80 0x7efe3c141160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.648 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.621+0000 7efe4179d640 10 monclient: _finish_hunting 0 2023-10-30T12:17:05.648 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.621+0000 7efe4179d640 1 monclient: found mon.noname-c 2023-10-30T12:17:05.648 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.621+0000 7efe4179d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:05.648 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.621+0000 7efe4179d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:17:05.649 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.621+0000 7efe4179d640 1 -- 172.21.15.150:0/1692711410 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c1da070 con 0x7efe3c0b0700 2023-10-30T12:17:05.649 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.624+0000 7efe4179d640 1 --2- 172.21.15.150:0/1692711410 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0b0700 0x7efe3c1d9b30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efe3400cb00 tx=0x7efe34002130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:05.649 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.624+0000 7efe397fa640 10 client.?.objecter ms_handle_connect 0x7efe3c0b0700 2023-10-30T12:17:05.649 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.624+0000 7efe397fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:05.650 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:05.632+0000 7f4e91df1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T12:17:05.651 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 1 -- 172.21.15.150:0/1692711410 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3400ab70 con 0x7efe3c0b0700 2023-10-30T12:17:05.651 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:05.652 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:05.652 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 10 monclient: dump: 2023-10-30T12:17:05.652 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:05.652 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:05.652 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:05.653 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:05.653 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:05.653 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:05.653 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:05.653 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:05.654 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:05.654 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:05.654 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 10 monclient: _finish_auth 0 2023-10-30T12:17:05.654 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 10 monclient: _check_auth_tickets 2023-10-30T12:17:05.654 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:05.655 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 1 -- 172.21.15.150:0/1692711410 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3400e040 con 0x7efe3c0b0700 2023-10-30T12:17:05.655 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:05.655 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 1 -- 172.21.15.150:0/1692711410 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe34012400 con 0x7efe3c0b0700 2023-10-30T12:17:05.655 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:05.655 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:05.655 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe397fa640 10 monclient: dump: 2023-10-30T12:17:05.656 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:05.656 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:05.656 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:05.656 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:05.656 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:05.657 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:05.657 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:05.657 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:05.657 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:05.657 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:05.658 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe3affd640 4 set_mon_vals no callback set 2023-10-30T12:17:05.658 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe41f9e640 5 monclient: authenticate success, global_id 4112 2023-10-30T12:17:05.658 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe41f9e640 10 monclient: _renew_subs 2023-10-30T12:17:05.658 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe41f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:05.658 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.628+0000 7efe41f9e640 1 -- 172.21.15.150:0/1692711410 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3c1c8ef0 con 0x7efe3c0b0700 2023-10-30T12:17:05.658 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.629+0000 7efe41f9e640 10 client.4112.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:05.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.629+0000 7efe41f9e640 10 monclient: _renew_subs 2023-10-30T12:17:05.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.629+0000 7efe41f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:05.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.629+0000 7efe41f9e640 1 -- 172.21.15.150:0/1692711410 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3c1d6810 con 0x7efe3c0b0700 2023-10-30T12:17:05.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.629+0000 7efe41f9e640 1 librados: init done 2023-10-30T12:17:05.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.629+0000 7efe41f9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.629+0000 7efe41f9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.629+0000 7efe41f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.629+0000 7efe41f9e640 1 -- 172.21.15.150:0/1692711410 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3c1d6cc0 con 0x7efe3c0b0700 2023-10-30T12:17:05.687 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:05.685+0000 7f4e91df1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T12:17:05.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.726+0000 7efe397fa640 1 -- 172.21.15.150:0/1692711410 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efe34012870 con 0x7efe3c0b0700 2023-10-30T12:17:05.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.734+0000 7efe397fa640 1 -- 172.21.15.150:0/1692711410 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7efe3401cb80 con 0x7efe3c0b0700 2023-10-30T12:17:05.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.734+0000 7efe397fa640 10 client.4112.objecter ms_dispatch 0x7efe3c000b90 osd_map(2..2 src has 1..2) v4 2023-10-30T12:17:05.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.734+0000 7efe397fa640 3 client.4112.objecter handle_osd_map got epochs [2,2] > 0 2023-10-30T12:17:05.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.734+0000 7efe397fa640 3 client.4112.objecter handle_osd_map decoding full epoch 2 2023-10-30T12:17:05.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.734+0000 7efe397fa640 20 client.4112.objecter dump_active .. 0 homeless 2023-10-30T12:17:05.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.734+0000 7efe397fa640 1 -- 172.21.15.150:0/1692711410 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7efe34007070 con 0x7efe3c0b0700 2023-10-30T12:17:05.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.734+0000 7efe397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:05.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.734+0000 7efe397fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:05.737 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:05.735+0000 7f4e91df1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T12:17:05.790 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:05.788+0000 7f4e91df1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T12:17:05.805 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.803+0000 7efe41f9e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "183f5799-ff32-4ada-b4ff-fd8fcbd82d08", "id": 0}] 2023-10-30T12:17:05.805 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.803+0000 7efe41f9e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "183f5799-ff32-4ada-b4ff-fd8fcbd82d08", "id": 0}] 2023-10-30T12:17:05.805 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.803+0000 7efe41f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:05.805 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.803+0000 7efe41f9e640 1 -- 172.21.15.150:0/1692711410 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd new", "uuid": "183f5799-ff32-4ada-b4ff-fd8fcbd82d08", "id": 0} v 0) v1 -- 0x7efe3c0b3370 con 0x7efe3c0b0700 2023-10-30T12:17:05.834 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:05.833+0000 7f4e91df1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T12:17:05.941 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:05.939+0000 7f4e91df1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-30T12:17:05.963 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.962+0000 7efe397fa640 1 -- 172.21.15.150:0/1692711410 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "183f5799-ff32-4ada-b4ff-fd8fcbd82d08", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7efe34007220 con 0x7efe3c0b0700 2023-10-30T12:17:05.963 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.962+0000 7efe397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "183f5799-ff32-4ada-b4ff-fd8fcbd82d08", "id": 0}] 2023-10-30T12:17:05.964 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.962+0000 7efe397fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:05.964 INFO:teuthology.orchestra.run.smithi150.stdout:0 2023-10-30T12:17:05.965 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 10 librados: watch_flush enter 2023-10-30T12:17:05.966 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 10 librados: watch_flush exit 2023-10-30T12:17:05.966 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 20 client.4112.objecter shutdown clearing up homeless session... 2023-10-30T12:17:05.966 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 10 client.4112.objecter successfully canceled tick 2023-10-30T12:17:05.966 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 10 monclient: shutdown 2023-10-30T12:17:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 1 -- 172.21.15.150:0/1692711410 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0b0700 msgr2=0x7efe3c1d9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 1 --2- 172.21.15.150:0/1692711410 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0b0700 0x7efe3c1d9b30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efe3400cb00 tx=0x7efe34002130 comp rx=0 tx=0).stop 2023-10-30T12:17:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 1 -- 172.21.15.150:0/1692711410 shutdown_connections 2023-10-30T12:17:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 1 --2- 172.21.15.150:0/1692711410 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe3c0b0700 0x7efe3c1d9b30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 1 --2- 172.21.15.150:0/1692711410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c0a9360 0x7efe3c1d6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 1 --2- 172.21.15.150:0/1692711410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3c0a8f80 0x7efe3c141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 1 -- 172.21.15.150:0/1692711410 >> 172.21.15.150:0/1692711410 conn(0x7efe3c0b7180 msgr2=0x7efe3c1cbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 1 -- 172.21.15.150:0/1692711410 shutdown_connections 2023-10-30T12:17:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 1 -- 172.21.15.150:0/1692711410 wait complete. 2023-10-30T12:17:05.969 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:05.963+0000 7efe41f9e640 1 librados: shutdown 2023-10-30T12:17:05.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new ee28bf60-9f29-4fb1-87a0-7e0707b143b4 1 2023-10-30T12:17:06.019 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:06.017+0000 7f4e91df1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T12:17:06.069 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:06.068+0000 7f4e91df1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T12:17:06.093 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.092+0000 7fc051439640 10 monclient: get_monmap_and_config 2023-10-30T12:17:06.094 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.092+0000 7fc051439640 10 monclient: build_initial_monmap 2023-10-30T12:17:06.094 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.092+0000 7fc051439640 10 monclient: monmap: 2023-10-30T12:17:06.094 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:06.095 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.095 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:17:06.093195+0000 2023-10-30T12:17:06.095 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:17:06.093195+0000 2023-10-30T12:17:06.095 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:06.096 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.096 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:06.096 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:06.097 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:06.097 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:06.097 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:06.097 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.099 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.092+0000 7fc051439640 1 Processor -- start 2023-10-30T12:17:06.099 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.092+0000 7fc051439640 1 -- start start 2023-10-30T12:17:06.100 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.092+0000 7fc051439640 10 monclient: init 2023-10-30T12:17:06.100 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:06.100 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-10-30T12:17:06.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc04c0b7c10 0x7fc04c0a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 10 monclient(hunting): picked mon.noname-c con 0x7fc04c0b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:06.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc04c141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:06.102 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc04c13e730 addr v1:172.21.15.148:6789/0 2023-10-30T12:17:06.102 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c0a9350 con 0x7fc04c13e730 2023-10-30T12:17:06.102 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c0aa6a0 con 0x7fc04c141940 2023-10-30T12:17:06.103 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.103 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04c0670a0 con 0x7fc04c0b7c10 2023-10-30T12:17:06.103 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:06.103 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.093+0000 7fc051439640 10 monclient(hunting): authenticate will time out at 1522.324516s 2023-10-30T12:17:06.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.100+0000 7fc04b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc04c13e730 0x7fc04c13eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.150:47542/0 (socket says 172.21.15.150:47542) 2023-10-30T12:17:06.105 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.100+0000 7fc04b7fe640 1 -- 172.21.15.150:0/615306446 learned_addr learned my addr 172.21.15.150:0/615306446 (peer_addr_for_me v1:172.21.15.150:0/0) 2023-10-30T12:17:06.105 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.103+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755733734 0 0) 0x7fc044001490 con 0x7fc04c13e730 2023-10-30T12:17:06.105 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.103+0000 7fc049ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:17:06.106 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.103+0000 7fc049ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.106 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.103+0000 7fc049ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.106 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.103+0000 7fc049ffb640 10 monclient(hunting): my global_id is 4116 2023-10-30T12:17:06.106 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.103+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc034003520 con 0x7fc04c13e730 2023-10-30T12:17:06.107 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.104+0000 7fc04affd640 1 --2- 172.21.15.150:0/615306446 >> v2:172.21.15.150:3300/0 conn(0x7fc04c0b7c10 0x7fc04c0a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.108 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.105+0000 7fc04affd640 10 monclient(hunting): get_auth_request con 0x7fc04c0b7c10 auth_method 0 2023-10-30T12:17:06.108 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.105+0000 7fc04affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.108 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.105+0000 7fc04affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.109 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.105+0000 7fc04affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.109 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.106+0000 7fc04affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.109 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.106+0000 7fc04affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.110 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.106+0000 7fc04affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:06.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3730946016 0 0) 0x7fc044001890 con 0x7fc04c13e730 2023-10-30T12:17:06.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 >> v2:172.21.15.150:3300/0 conn(0x7fc04c0b7c10 msgr2=0x7fc04c0a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:06.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 --2- 172.21.15.150:0/615306446 >> v2:172.21.15.150:3300/0 conn(0x7fc04c0b7c10 0x7fc04c0a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.115 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 >> v1:172.21.15.150:6789/0 conn(0x7fc04c141940 legacy=0x7fc04c141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:06.115 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient: _finish_hunting 0 2023-10-30T12:17:06.115 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:17:06.115 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.148:6789/0 2023-10-30T12:17:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c0aa910 con 0x7fc04c13e730 2023-10-30T12:17:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient: _finish_auth 0 2023-10-30T12:17:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:17:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1567732308 0 0) 0x7fc044001d30 con 0x7fc04c13e730 2023-10-30T12:17:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc051439640 5 monclient: authenticate success, global_id 4116 2023-10-30T12:17:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc051439640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:17:06.119 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient: dump: 2023-10-30T12:17:06.119 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:06.119 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.119 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.121 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:06.121 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:06.121 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:06.121 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.122 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] but i'm connected to v1:172.21.15.148:6789/0, reconnecting 2023-10-30T12:17:06.122 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:06.122 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 >> v1:172.21.15.148:6789/0 conn(0x7fc04c13e730 legacy=0x7fc04c13eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:17:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 --2- 172.21.15.150:0/615306446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc0340039c0 0x7fc034003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient(hunting): picked mon.a con 0x7fc0340039c0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 --2- 172.21.15.150:0/615306446 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc0340047c0 0x7fc034004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient(hunting): picked mon.c con 0x7fc0340047c0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 --2- 172.21.15.150:0/615306446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0340051d0 0x7fc03400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient(hunting): picked mon.b con 0x7fc0340051d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:06.125 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.125 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c0670a0 con 0x7fc0340039c0 2023-10-30T12:17:06.125 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.125 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03400c120 con 0x7fc0340051d0 2023-10-30T12:17:06.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03400c2f0 con 0x7fc0340047c0 2023-10-30T12:17:06.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:06.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc049ffb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:17:06.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc051439640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:06.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.108+0000 7fc04a7fc640 1 --2- 172.21.15.150:0/615306446 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc0340047c0 0x7fc034004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.109+0000 7fc04a7fc640 10 monclient(hunting): get_auth_request con 0x7fc0340047c0 auth_method 0 2023-10-30T12:17:06.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.109+0000 7fc04a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.109+0000 7fc04a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.109+0000 7fc04a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:06.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.109+0000 7fc04a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.109+0000 7fc04a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.109+0000 7fc04a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:06.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 10 monclient(hunting): handle_auth_done global_id 4116 payload 293 2023-10-30T12:17:06.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 1 -- 172.21.15.150:0/615306446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0340051d0 msgr2=0x7fc03400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:06.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 1 --2- 172.21.15.150:0/615306446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0340051d0 0x7fc03400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 1 -- 172.21.15.150:0/615306446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc0340039c0 msgr2=0x7fc034003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:06.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 1 --2- 172.21.15.150:0/615306446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc0340039c0 0x7fc034003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:17:06.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 1 monclient: found mon.c 2023-10-30T12:17:06.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:06.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:06.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04b7fe640 1 -- 172.21.15.150:0/615306446 reap_dead start 2023-10-30T12:17:06.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 1 -- 172.21.15.150:0/615306446 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc044001890 con 0x7fc0340047c0 2023-10-30T12:17:06.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:17:06.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:17:06.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.111+0000 7fc04a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:06.134 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:06.130+0000 7f4e91df1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T12:17:06.139 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.114+0000 7fc04a7fc640 1 --2- 172.21.15.150:0/615306446 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc0340047c0 0x7fc034004c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc040001720 tx=0x7fc0340057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18eb0d537bc3efd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:06.139 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.117+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04002e050 con 0x7fc0340047c0 2023-10-30T12:17:06.140 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.117+0000 7fc049ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:06.140 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.117+0000 7fc049ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:06.140 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.117+0000 7fc049ffb640 10 monclient: dump: 2023-10-30T12:17:06.140 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:06.140 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.141 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.141 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.141 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:06.141 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.141 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:06.142 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:06.142 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:06.142 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.142 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.117+0000 7fc051439640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:06.142 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.119+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0400052d0 con 0x7fc0340047c0 2023-10-30T12:17:06.143 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.119+0000 7fc049ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:06.143 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.119+0000 7fc049ffb640 1 -- 172.21.15.150:0/615306446 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040003830 con 0x7fc0340047c0 2023-10-30T12:17:06.143 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.119+0000 7fc051439640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:06.143 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.119+0000 7fc051439640 4 set_mon_vals no callback set 2023-10-30T12:17:06.143 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.119+0000 7fc049ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:06.144 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.119+0000 7fc049ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:06.144 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.119+0000 7fc049ffb640 10 monclient: dump: 2023-10-30T12:17:06.144 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:06.144 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.144 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.145 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.145 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:06.145 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.145 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:06.145 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:06.146 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:06.146 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.146 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.120+0000 7fc051439640 10 monclient: shutdown 2023-10-30T12:17:06.146 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.120+0000 7fc051439640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:06.146 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.120+0000 7fc051439640 1 -- 172.21.15.150:0/615306446 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc0340047c0 msgr2=0x7fc034004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:06.146 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.120+0000 7fc051439640 1 --2- 172.21.15.150:0/615306446 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc0340047c0 0x7fc034004c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc040001720 tx=0x7fc0340057f0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.147 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.120+0000 7fc051439640 1 -- 172.21.15.150:0/615306446 shutdown_connections 2023-10-30T12:17:06.147 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.120+0000 7fc051439640 1 --2- 172.21.15.150:0/615306446 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc0340047c0 0x7fc034004c40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.147 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.120+0000 7fc051439640 1 -- 172.21.15.150:0/615306446 >> 172.21.15.150:0/615306446 conn(0x7fc04c06c660 msgr2=0x7fc04c0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:06.147 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.120+0000 7fc051439640 1 -- 172.21.15.150:0/615306446 shutdown_connections 2023-10-30T12:17:06.147 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.120+0000 7fc051439640 1 -- 172.21.15.150:0/615306446 wait complete. 2023-10-30T12:17:06.148 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.121+0000 7fc051439640 10 monclient: build_initial_monmap 2023-10-30T12:17:06.148 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.121+0000 7fc051439640 10 monclient: monmap: 2023-10-30T12:17:06.148 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:06.148 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:06.148 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2023-10-30T12:17:06.148 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2023-10-30T12:17:06.149 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:06.149 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2023-10-30T12:17:06.149 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:06.149 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:06.149 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:06.150 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.150 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.121+0000 7fc051439640 1 librados: starting msgr at 2023-10-30T12:17:06.150 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.121+0000 7fc051439640 1 librados: starting objecter 2023-10-30T12:17:06.150 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 1 Processor -- start 2023-10-30T12:17:06.150 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 1 -- start start 2023-10-30T12:17:06.151 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 1 librados: setting wanted keys 2023-10-30T12:17:06.151 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 1 librados: calling monclient init 2023-10-30T12:17:06.151 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient: init 2023-10-30T12:17:06.151 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:06.151 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:17:06.152 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0340047c0 0x7fc04c1da7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.152 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient(hunting): picked mon.noname-b con 0x7fc0340047c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:06.152 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc04c1dace0 0x7fc04c1cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.152 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient(hunting): picked mon.noname-a con 0x7fc04c1dace0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:06.152 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc04c0aadf0 0x7fc04c1cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient(hunting): picked mon.noname-c con 0x7fc04c0aadf0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:06.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c1425c0 con 0x7fc04c1dace0 2023-10-30T12:17:06.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c1db220 con 0x7fc0340047c0 2023-10-30T12:17:06.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.154 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c1cbae0 con 0x7fc04c0aadf0 2023-10-30T12:17:06.154 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:06.154 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.122+0000 7fc051439640 10 monclient(hunting): authenticate will time out at 1522.353805s 2023-10-30T12:17:06.154 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.124+0000 7fc04a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc04c1dace0 0x7fc04c1cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.154 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0340047c0 0x7fc04c1da7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.155 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc04c1dace0 0x7fc04c1cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.150:38560/0 (socket says 172.21.15.150:38560) 2023-10-30T12:17:06.155 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04a7fc640 1 -- 172.21.15.150:0/3913697888 learned_addr learned my addr 172.21.15.150:0/3913697888 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:06.155 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04a7fc640 10 monclient(hunting): get_auth_request con 0x7fc04c1dace0 auth_method 0 2023-10-30T12:17:06.155 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.155 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.156 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.156 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04b7fe640 1 --2- 172.21.15.150:0/3913697888 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc04c0aadf0 0x7fc04c1cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.156 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04affd640 10 monclient(hunting): get_auth_request con 0x7fc0340047c0 auth_method 0 2023-10-30T12:17:06.156 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.156 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.125+0000 7fc04affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.126+0000 7fc04affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.126+0000 7fc04affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.126+0000 7fc04affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:06.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.126+0000 7fc04b7fe640 10 monclient(hunting): get_auth_request con 0x7fc04c0aadf0 auth_method 0 2023-10-30T12:17:06.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.126+0000 7fc04b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.158 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.126+0000 7fc04b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.158 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.126+0000 7fc04b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.158 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.127+0000 7fc04a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.158 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.127+0000 7fc04a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.158 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.127+0000 7fc04a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:06.159 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.127+0000 7fc04b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.159 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.127+0000 7fc04b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.159 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.127+0000 7fc04b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:06.159 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.129+0000 7fc04affd640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2023-10-30T12:17:06.159 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.129+0000 7fc04affd640 1 -- 172.21.15.150:0/3913697888 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc04c0aadf0 msgr2=0x7fc04c1cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:06.160 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.129+0000 7fc04affd640 1 --2- 172.21.15.150:0/3913697888 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc04c0aadf0 0x7fc04c1cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.160 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.129+0000 7fc04affd640 1 -- 172.21.15.150:0/3913697888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc04c1dace0 msgr2=0x7fc04c1cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:06.160 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.129+0000 7fc04affd640 1 --2- 172.21.15.150:0/3913697888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc04c1dace0 0x7fc04c1cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.160 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.129+0000 7fc04affd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:06.160 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.129+0000 7fc04affd640 1 monclient: found mon.noname-b 2023-10-30T12:17:06.160 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.129+0000 7fc04affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:06.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.129+0000 7fc04affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.129+0000 7fc04affd640 1 -- 172.21.15.150:0/3913697888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c1cd230 con 0x7fc0340047c0 2023-10-30T12:17:06.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.131+0000 7fc04affd640 1 --2- 172.21.15.150:0/3913697888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0340047c0 0x7fc04c1da7a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00e820 tx=0x7fc03c00ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:06.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.131+0000 7fc0337fe640 10 client.?.objecter ms_handle_connect 0x7fc0340047c0 2023-10-30T12:17:06.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.131+0000 7fc0337fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:06.162 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 1 -- 172.21.15.150:0/3913697888 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c00ee70 con 0x7fc0340047c0 2023-10-30T12:17:06.162 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:06.162 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:06.162 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 10 monclient: dump: 2023-10-30T12:17:06.162 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:06.163 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.163 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.163 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.163 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:06.163 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.163 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:06.164 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:06.165 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:06.165 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 1 -- 172.21.15.150:0/3913697888 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c00d040 con 0x7fc0340047c0 2023-10-30T12:17:06.165 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc051439640 5 monclient: authenticate success, global_id 4117 2023-10-30T12:17:06.165 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc051439640 10 monclient: _renew_subs 2023-10-30T12:17:06.165 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc051439640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc051439640 1 -- 172.21.15.150:0/3913697888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c1cd430 con 0x7fc0340047c0 2023-10-30T12:17:06.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:06.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 1 -- 172.21.15.150:0/3913697888 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c00cb20 con 0x7fc0340047c0 2023-10-30T12:17:06.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:06.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:06.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0337fe640 10 monclient: dump: 2023-10-30T12:17:06.167 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:06.167 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.167 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.167 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.167 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:06.168 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.168 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:06.168 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:06.168 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:06.168 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.169 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc0497fa640 4 set_mon_vals no callback set 2023-10-30T12:17:06.169 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc051439640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:06.169 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc051439640 10 monclient: _renew_subs 2023-10-30T12:17:06.169 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc051439640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.169 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc051439640 1 -- 172.21.15.150:0/3913697888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c1d7210 con 0x7fc0340047c0 2023-10-30T12:17:06.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.136+0000 7fc051439640 1 librados: init done 2023-10-30T12:17:06.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.137+0000 7fc051439640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:06.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.137+0000 7fc051439640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:06.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.137+0000 7fc051439640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.137+0000 7fc051439640 1 -- 172.21.15.150:0/3913697888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c13e4c0 con 0x7fc0340047c0 2023-10-30T12:17:06.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.160+0000 7fc0337fe640 1 -- 172.21.15.150:0/3913697888 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc03c004020 con 0x7fc0340047c0 2023-10-30T12:17:06.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.160+0000 7fc0337fe640 1 -- 172.21.15.150:0/3913697888 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fc03c003720 con 0x7fc0340047c0 2023-10-30T12:17:06.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.160+0000 7fc0337fe640 10 client.4117.objecter ms_dispatch 0x7fc04c000b90 osd_map(3..3 src has 1..3) v4 2023-10-30T12:17:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.160+0000 7fc0337fe640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2023-10-30T12:17:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.160+0000 7fc0337fe640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2023-10-30T12:17:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.160+0000 7fc0337fe640 20 client.4117.objecter dump_active .. 0 homeless 2023-10-30T12:17:06.179 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:06.178+0000 7f4e91df1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T12:17:06.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.331+0000 7fc0337fe640 1 -- 172.21.15.150:0/3913697888 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc03c0042d0 con 0x7fc0340047c0 2023-10-30T12:17:06.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.331+0000 7fc0337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:06.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.331+0000 7fc0337fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:06.461 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.460+0000 7fc051439640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ee28bf60-9f29-4fb1-87a0-7e0707b143b4", "id": 1}] 2023-10-30T12:17:06.462 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.460+0000 7fc051439640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ee28bf60-9f29-4fb1-87a0-7e0707b143b4", "id": 1}] 2023-10-30T12:17:06.462 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.460+0000 7fc051439640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.462 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.460+0000 7fc051439640 1 -- 172.21.15.150:0/3913697888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ee28bf60-9f29-4fb1-87a0-7e0707b143b4", "id": 1} v 0) v1 -- 0x7fc04c0aa350 con 0x7fc0340047c0 2023-10-30T12:17:06.553 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:06.552+0000 7fc4f022e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T12:17:06.564 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:06.562+0000 7f4e91df1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-30T12:17:06.584 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.582+0000 7fc0337fe640 1 -- 172.21.15.150:0/3913697888 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ee28bf60-9f29-4fb1-87a0-7e0707b143b4", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc03c005990 con 0x7fc0340047c0 2023-10-30T12:17:06.584 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.582+0000 7fc0337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ee28bf60-9f29-4fb1-87a0-7e0707b143b4", "id": 1}] 2023-10-30T12:17:06.584 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.582+0000 7fc0337fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:06.585 INFO:teuthology.orchestra.run.smithi150.stdout:1 2023-10-30T12:17:06.586 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 10 librados: watch_flush enter 2023-10-30T12:17:06.586 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 10 librados: watch_flush exit 2023-10-30T12:17:06.586 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 20 client.4117.objecter shutdown clearing up homeless session... 2023-10-30T12:17:06.586 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 10 client.4117.objecter successfully canceled tick 2023-10-30T12:17:06.587 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 10 monclient: shutdown 2023-10-30T12:17:06.587 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:06.587 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 1 -- 172.21.15.150:0/3913697888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0340047c0 msgr2=0x7fc04c1da7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:06.587 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 1 --2- 172.21.15.150:0/3913697888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0340047c0 0x7fc04c1da7a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00e820 tx=0x7fc03c00ec20 comp rx=0 tx=0).stop 2023-10-30T12:17:06.587 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 1 -- 172.21.15.150:0/3913697888 shutdown_connections 2023-10-30T12:17:06.588 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 1 --2- 172.21.15.150:0/3913697888 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc04c0aadf0 0x7fc04c1cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.588 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 1 --2- 172.21.15.150:0/3913697888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc04c1dace0 0x7fc04c1cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.588 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 1 --2- 172.21.15.150:0/3913697888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0340047c0 0x7fc04c1da7a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.588 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 1 -- 172.21.15.150:0/3913697888 >> 172.21.15.150:0/3913697888 conn(0x7fc04c06c660 msgr2=0x7fc04c0a84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:06.588 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 1 -- 172.21.15.150:0/3913697888 shutdown_connections 2023-10-30T12:17:06.589 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 1 -- 172.21.15.150:0/3913697888 wait complete. 2023-10-30T12:17:06.589 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.584+0000 7fc051439640 1 librados: shutdown 2023-10-30T12:17:06.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 22573689-d164-4530-aa70-adbab2232e1e 2 2023-10-30T12:17:06.612 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:06.611+0000 7fc4f022e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T12:17:06.699 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:06.698+0000 7f4e91df1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T12:17:06.701 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.699+0000 7fc373778640 10 monclient: get_monmap_and_config 2023-10-30T12:17:06.701 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.700+0000 7fc373778640 10 monclient: build_initial_monmap 2023-10-30T12:17:06.701 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.700+0000 7fc373778640 10 monclient: monmap: 2023-10-30T12:17:06.701 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:17:06.700903+0000 2023-10-30T12:17:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:17:06.700903+0000 2023-10-30T12:17:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.703 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:06.703 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:06.703 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:06.703 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:06.703 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:06.704 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.706 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.700+0000 7fc373778640 1 Processor -- start 2023-10-30T12:17:06.706 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.700+0000 7fc373778640 1 -- start start 2023-10-30T12:17:06.706 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.700+0000 7fc373778640 10 monclient: init 2023-10-30T12:17:06.707 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:06.707 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-10-30T12:17:06.707 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc36c0b7c10 0x7fc36c0a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.707 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 10 monclient(hunting): picked mon.noname-c con 0x7fc36c0b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:06.707 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc36c141940 0x7fc36c141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.707 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 10 monclient(hunting): picked mon.noname-b con 0x7fc36c141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:06.708 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc36c0b1c10 0x7fc36c0b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.708 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 10 monclient(hunting): picked mon.noname-a con 0x7fc36c0b1c10 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:06.708 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.708 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36c0670a0 con 0x7fc36c0b1c10 2023-10-30T12:17:06.708 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.709 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36c0b7ff0 con 0x7fc36c0b7c10 2023-10-30T12:17:06.709 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.709 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0a9350 con 0x7fc36c141940 2023-10-30T12:17:06.709 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:06.709 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc373778640 10 monclient(hunting): authenticate will time out at 1522.932347s 2023-10-30T12:17:06.710 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.701+0000 7fc371514640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc36c0b7c10 0x7fc36c0a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.710 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc370d13640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc36c141940 0x7fc36c141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.710 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371d15640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc36c0b1c10 0x7fc36c0b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.710 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371514640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc36c0b7c10 0x7fc36c0a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38638/0 (socket says 172.21.15.150:38638) 2023-10-30T12:17:06.710 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371514640 1 -- 172.21.15.150:0/3891938102 learned_addr learned my addr 172.21.15.150:0/3891938102 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:06.710 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371514640 10 monclient(hunting): get_auth_request con 0x7fc36c0b7c10 auth_method 0 2023-10-30T12:17:06.711 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.711 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371514640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.711 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371514640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.711 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc370d13640 10 monclient(hunting): get_auth_request con 0x7fc36c141940 auth_method 0 2023-10-30T12:17:06.712 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc370d13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.712 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc370d13640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.712 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc370d13640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.712 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371d15640 10 monclient(hunting): get_auth_request con 0x7fc36c0b1c10 auth_method 0 2023-10-30T12:17:06.712 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.713 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371d15640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.713 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371d15640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.713 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371514640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.713 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371514640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.713 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371514640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:06.713 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc370d13640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.714 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc370d13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.714 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc370d13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:06.714 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.714 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.702+0000 7fc371d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.715 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.703+0000 7fc371d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:06.715 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.704+0000 7fc371514640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2023-10-30T12:17:06.715 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.704+0000 7fc371514640 1 -- 172.21.15.150:0/3891938102 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc36c141940 msgr2=0x7fc36c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:06.715 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.704+0000 7fc371514640 1 --2- 172.21.15.150:0/3891938102 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc36c141940 0x7fc36c141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.715 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.704+0000 7fc371514640 1 -- 172.21.15.150:0/3891938102 >> v2:172.21.15.148:3300/0 conn(0x7fc36c0b1c10 msgr2=0x7fc36c0b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:06.716 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.704+0000 7fc371514640 1 --2- 172.21.15.150:0/3891938102 >> v2:172.21.15.148:3300/0 conn(0x7fc36c0b1c10 0x7fc36c0b2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.716 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.704+0000 7fc371514640 10 monclient: _finish_hunting 0 2023-10-30T12:17:06.716 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.704+0000 7fc371514640 1 monclient: found mon.noname-c 2023-10-30T12:17:06.716 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.704+0000 7fc371514640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:06.716 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.704+0000 7fc371514640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.717 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.704+0000 7fc371514640 1 -- 172.21.15.150:0/3891938102 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0aa6a0 con 0x7fc36c0b7c10 2023-10-30T12:17:06.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.706+0000 7fc371514640 1 --2- 172.21.15.150:0/3891938102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 0x7fc36c0a8aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0b4d80 tx=0x7fc35c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26cd9ef06fc5ae1f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:06.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 1 -- 172.21.15.150:0/3891938102 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0317e0 con 0x7fc36c0b7c10 2023-10-30T12:17:06.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:06.722 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:06.722 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 10 monclient: dump: 2023-10-30T12:17:06.722 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:06.722 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.722 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.723 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.723 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:06.723 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.723 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:06.723 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:06.724 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:06.724 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 10 monclient: _finish_auth 0 2023-10-30T12:17:06.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 10 monclient: _check_auth_tickets 2023-10-30T12:17:06.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:06.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 1 -- 172.21.15.150:0/3891938102 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c031950 con 0x7fc36c0b7c10 2023-10-30T12:17:06.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc373778640 5 monclient: authenticate success, global_id 4120 2023-10-30T12:17:06.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc373778640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:06.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:06.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 1 -- 172.21.15.150:0/3891938102 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c031cf0 con 0x7fc36c0b7c10 2023-10-30T12:17:06.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc373778640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc373778640 4 set_mon_vals no callback set 2023-10-30T12:17:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.709+0000 7fc363fff640 10 monclient: dump: 2023-10-30T12:17:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:06.728 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 10 monclient: shutdown 2023-10-30T12:17:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 1 -- 172.21.15.150:0/3891938102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 msgr2=0x7fc36c0a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 1 --2- 172.21.15.150:0/3891938102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 0x7fc36c0a8aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0b4d80 tx=0x7fc35c02b880 comp rx=0 tx=0).stop 2023-10-30T12:17:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 1 -- 172.21.15.150:0/3891938102 shutdown_connections 2023-10-30T12:17:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 1 --2- 172.21.15.150:0/3891938102 >> v2:172.21.15.148:3300/0 conn(0x7fc36c0b1c10 0x7fc36c0b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 1 --2- 172.21.15.150:0/3891938102 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc36c141940 0x7fc36c141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 1 --2- 172.21.15.150:0/3891938102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 0x7fc36c0a8aa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 1 -- 172.21.15.150:0/3891938102 >> 172.21.15.150:0/3891938102 conn(0x7fc36c06c660 msgr2=0x7fc36c0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 1 -- 172.21.15.150:0/3891938102 shutdown_connections 2023-10-30T12:17:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.710+0000 7fc373778640 1 -- 172.21.15.150:0/3891938102 wait complete. 2023-10-30T12:17:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.711+0000 7fc373778640 10 monclient: build_initial_monmap 2023-10-30T12:17:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.711+0000 7fc373778640 10 monclient: monmap: 2023-10-30T12:17:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2023-10-30T12:17:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2023-10-30T12:17:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:06.732 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2023-10-30T12:17:06.732 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:06.732 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:06.732 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:06.732 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.732 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.711+0000 7fc373778640 1 librados: starting msgr at 2023-10-30T12:17:06.733 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.711+0000 7fc373778640 1 librados: starting objecter 2023-10-30T12:17:06.733 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.711+0000 7fc373778640 1 Processor -- start 2023-10-30T12:17:06.733 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.711+0000 7fc373778640 1 -- start start 2023-10-30T12:17:06.733 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.711+0000 7fc373778640 1 librados: setting wanted keys 2023-10-30T12:17:06.733 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.711+0000 7fc373778640 1 librados: calling monclient init 2023-10-30T12:17:06.734 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.711+0000 7fc373778640 10 monclient: init 2023-10-30T12:17:06.734 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:06.734 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:17:06.734 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c0b1c10 0x7fc36c0b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.734 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 10 monclient(hunting): picked mon.noname-a con 0x7fc36c0b1c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:06.734 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 0x7fc36c0adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 10 monclient(hunting): picked mon.noname-b con 0x7fc36c0b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:06.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc36c141940 0x7fc36c0ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:06.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 10 monclient(hunting): picked mon.noname-c con 0x7fc36c141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:06.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0b18a0 con 0x7fc36c0b1c10 2023-10-30T12:17:06.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c1c9300 con 0x7fc36c0b7c10 2023-10-30T12:17:06.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:06.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c1c7ee0 con 0x7fc36c141940 2023-10-30T12:17:06.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:06.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc373778640 10 monclient(hunting): authenticate will time out at 1522.943479s 2023-10-30T12:17:06.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.712+0000 7fc370d13640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 0x7fc36c0adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371d15640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc36c141940 0x7fc36c0ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371514640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c0b1c10 0x7fc36c0b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:06.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc370d13640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 0x7fc36c0adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38654/0 (socket says 172.21.15.150:38654) 2023-10-30T12:17:06.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc370d13640 1 -- 172.21.15.150:0/393847549 learned_addr learned my addr 172.21.15.150:0/393847549 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:06.738 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc370d13640 10 monclient(hunting): get_auth_request con 0x7fc36c0b7c10 auth_method 0 2023-10-30T12:17:06.738 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc370d13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.738 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc370d13640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.738 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc370d13640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.738 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371d15640 10 monclient(hunting): get_auth_request con 0x7fc36c141940 auth_method 0 2023-10-30T12:17:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371d15640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371d15640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371514640 10 monclient(hunting): get_auth_request con 0x7fc36c0b1c10 auth_method 0 2023-10-30T12:17:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371514640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371514640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc370d13640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc370d13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc370d13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:06.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371514640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:06.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.713+0000 7fc371514640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:06.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.714+0000 7fc371514640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:06.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.716+0000 7fc370d13640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2023-10-30T12:17:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.716+0000 7fc370d13640 1 -- 172.21.15.150:0/393847549 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc36c141940 msgr2=0x7fc36c0ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.716+0000 7fc370d13640 1 --2- 172.21.15.150:0/393847549 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc36c141940 0x7fc36c0ae4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.716+0000 7fc370d13640 1 -- 172.21.15.150:0/393847549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c0b1c10 msgr2=0x7fc36c0b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.716+0000 7fc370d13640 1 --2- 172.21.15.150:0/393847549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c0b1c10 0x7fc36c0b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.716+0000 7fc370d13640 10 monclient: _finish_hunting 0 2023-10-30T12:17:06.743 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.716+0000 7fc370d13640 1 monclient: found mon.noname-b 2023-10-30T12:17:06.743 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.716+0000 7fc370d13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:06.743 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.716+0000 7fc370d13640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.743 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.716+0000 7fc370d13640 1 -- 172.21.15.150:0/393847549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c1c9500 con 0x7fc36c0b7c10 2023-10-30T12:17:06.743 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.718+0000 7fc370d13640 1 --2- 172.21.15.150:0/393847549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 0x7fc36c0adf60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e9e0 tx=0x7fc36800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:06.743 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.718+0000 7fc361ffb640 10 client.?.objecter ms_handle_connect 0x7fc36c0b7c10 2023-10-30T12:17:06.744 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.718+0000 7fc361ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.721+0000 7fc361ffb640 1 -- 172.21.15.150:0/393847549 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36800c970 con 0x7fc36c0b7c10 2023-10-30T12:17:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.721+0000 7fc361ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.721+0000 7fc361ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc361ffb640 10 monclient: dump: 2023-10-30T12:17:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:06.748 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:06.748 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc361ffb640 10 monclient: _finish_auth 0 2023-10-30T12:17:06.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc361ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:17:06.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc361ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:06.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc361ffb640 1 -- 172.21.15.150:0/393847549 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36800d040 con 0x7fc36c0b7c10 2023-10-30T12:17:06.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc361ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:06.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc361ffb640 1 -- 172.21.15.150:0/393847549 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36800cd20 con 0x7fc36c0b7c10 2023-10-30T12:17:06.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc373778640 5 monclient: authenticate success, global_id 4123 2023-10-30T12:17:06.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc373778640 10 monclient: _renew_subs 2023-10-30T12:17:06.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc373778640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc373778640 1 -- 172.21.15.150:0/393847549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c0afe30 con 0x7fc36c0b7c10 2023-10-30T12:17:06.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc3637fe640 4 set_mon_vals no callback set 2023-10-30T12:17:06.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc361ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:06.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc361ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:06.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc361ffb640 10 monclient: dump: 2023-10-30T12:17:06.751 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:06.751 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:06.751 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.751 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:06.751 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:06.752 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:06.752 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:06.752 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:06.752 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:06.752 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:06.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc373778640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc373778640 10 monclient: _renew_subs 2023-10-30T12:17:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc373778640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc373778640 1 -- 172.21.15.150:0/393847549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c1c8370 con 0x7fc36c0b7c10 2023-10-30T12:17:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.722+0000 7fc373778640 1 librados: init done 2023-10-30T12:17:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.723+0000 7fc373778640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.723+0000 7fc373778640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.723+0000 7fc373778640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.723+0000 7fc373778640 1 -- 172.21.15.150:0/393847549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c0a97c0 con 0x7fc36c0b7c10 2023-10-30T12:17:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.726+0000 7fc361ffb640 1 -- 172.21.15.150:0/393847549 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc368005c90 con 0x7fc36c0b7c10 2023-10-30T12:17:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.726+0000 7fc361ffb640 1 -- 172.21.15.150:0/393847549 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fc3680067a0 con 0x7fc36c0b7c10 2023-10-30T12:17:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.726+0000 7fc361ffb640 10 client.4123.objecter ms_dispatch 0x7fc36c000b90 osd_map(4..4 src has 1..4) v4 2023-10-30T12:17:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.726+0000 7fc361ffb640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2023-10-30T12:17:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.726+0000 7fc361ffb640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2023-10-30T12:17:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.726+0000 7fc361ffb640 20 client.4123.objecter dump_active .. 0 homeless 2023-10-30T12:17:06.810 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:06.808+0000 7f4e91df1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T12:17:06.825 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.823+0000 7fc361ffb640 1 -- 172.21.15.150:0/393847549 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc368005f40 con 0x7fc36c0b7c10 2023-10-30T12:17:06.825 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.823+0000 7fc361ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:06.825 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.823+0000 7fc361ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:06.855 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:06.853+0000 7f4e91df1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T12:17:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.894+0000 7fc373778640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "22573689-d164-4530-aa70-adbab2232e1e", "id": 2}] 2023-10-30T12:17:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.894+0000 7fc373778640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "22573689-d164-4530-aa70-adbab2232e1e", "id": 2}] 2023-10-30T12:17:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.895+0000 7fc373778640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:06.897 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.895+0000 7fc373778640 1 -- 172.21.15.150:0/393847549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "22573689-d164-4530-aa70-adbab2232e1e", "id": 2} v 0) v1 -- 0x7fc36c0a8fb0 con 0x7fc36c0b7c10 2023-10-30T12:17:06.959 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:06.957+0000 7f4e91df1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:17:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.968+0000 7fc361ffb640 1 -- 172.21.15.150:0/393847549 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "22573689-d164-4530-aa70-adbab2232e1e", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc3680060f0 con 0x7fc36c0b7c10 2023-10-30T12:17:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.968+0000 7fc361ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "22573689-d164-4530-aa70-adbab2232e1e", "id": 2}] 2023-10-30T12:17:06.970 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.968+0000 7fc361ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:06.970 INFO:teuthology.orchestra.run.smithi150.stdout:2 2023-10-30T12:17:06.971 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 10 librados: watch_flush enter 2023-10-30T12:17:06.972 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 10 librados: watch_flush exit 2023-10-30T12:17:06.972 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 20 client.4123.objecter shutdown clearing up homeless session... 2023-10-30T12:17:06.972 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 10 client.4123.objecter successfully canceled tick 2023-10-30T12:17:06.972 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 10 monclient: shutdown 2023-10-30T12:17:06.972 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 1 -- 172.21.15.150:0/393847549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 msgr2=0x7fc36c0adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 1 --2- 172.21.15.150:0/393847549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 0x7fc36c0adf60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e9e0 tx=0x7fc36800eea0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 1 -- 172.21.15.150:0/393847549 shutdown_connections 2023-10-30T12:17:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 1 --2- 172.21.15.150:0/393847549 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc36c141940 0x7fc36c0ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 1 --2- 172.21.15.150:0/393847549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36c0b7c10 0x7fc36c0adf60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 1 --2- 172.21.15.150:0/393847549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c0b1c10 0x7fc36c0b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 1 -- 172.21.15.150:0/393847549 >> 172.21.15.150:0/393847549 conn(0x7fc36c06c660 msgr2=0x7fc36c13dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 1 -- 172.21.15.150:0/393847549 shutdown_connections 2023-10-30T12:17:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 1 -- 172.21.15.150:0/393847549 wait complete. 2023-10-30T12:17:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:06.969+0000 7fc373778640 1 librados: shutdown 2023-10-30T12:17:06.981 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 36c6a615-3a6a-4c78-86ed-546ddf04aa1f 3 2023-10-30T12:17:07.057 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:07.056+0000 7f4e91df1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T12:17:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.120+0000 7f0829800640 10 monclient: get_monmap_and_config 2023-10-30T12:17:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.120+0000 7f0829800640 10 monclient: build_initial_monmap 2023-10-30T12:17:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.120+0000 7f0829800640 10 monclient: monmap: 2023-10-30T12:17:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:17:07.121051+0000 2023-10-30T12:17:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:17:07.121051+0000 2023-10-30T12:17:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:07.124 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:07.124 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:07.124 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.125 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.120+0000 7f0829800640 1 Processor -- start 2023-10-30T12:17:07.125 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.120+0000 7f0829800640 1 -- start start 2023-10-30T12:17:07.125 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.120+0000 7f0829800640 10 monclient: init 2023-10-30T12:17:07.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:07.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-10-30T12:17:07.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f08240b7c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:07.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0824141940 0x7f0824141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 10 monclient(hunting): picked mon.noname-a con 0x7f0824141940 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:07.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f082413e730 0x7f082413eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 10 monclient(hunting): picked mon.noname-b con 0x7f082413e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:07.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08240a9350 con 0x7f08240b7c10 2023-10-30T12:17:07.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08240670a0 con 0x7f0824141940 2023-10-30T12:17:07.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08240b7ff0 con 0x7f082413e730 2023-10-30T12:17:07.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:07.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.121+0000 7f0829800640 10 monclient(hunting): authenticate will time out at 1523.352353s 2023-10-30T12:17:07.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.127+0000 7f0822ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f08240b7c10 0x7f08240a8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50602/0 (socket says 172.21.15.150:50602) 2023-10-30T12:17:07.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.127+0000 7f0822ffd640 1 -- 172.21.15.150:0/1623513088 learned_addr learned my addr 172.21.15.150:0/1623513088 (peer_addr_for_me v1:172.21.15.150:0/0) 2023-10-30T12:17:07.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.127+0000 7f08227fc640 1 --2- 172.21.15.150:0/1623513088 >> v2:172.21.15.148:3300/0 conn(0x7f0824141940 0x7f0824141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.128+0000 7f08227fc640 10 monclient(hunting): get_auth_request con 0x7f0824141940 auth_method 0 2023-10-30T12:17:07.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.128+0000 7f08227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.128+0000 7f08227fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.128+0000 7f08227fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.128+0000 7f08237fe640 1 --2- 172.21.15.150:0/1623513088 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f082413e730 0x7f082413eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.128+0000 7f08227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.128+0000 7f08227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.128+0000 7f08227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.129+0000 7f08237fe640 10 monclient(hunting): get_auth_request con 0x7f082413e730 auth_method 0 2023-10-30T12:17:07.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.129+0000 7f08237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.129+0000 7f08237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.129+0000 7f08237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.130+0000 7f08237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.130+0000 7f08237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.130+0000 7f08237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.134 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.131+0000 7f08227fc640 10 monclient(hunting): handle_auth_done global_id 4128 payload 274 2023-10-30T12:17:07.134 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.131+0000 7f08227fc640 1 -- 172.21.15.150:0/1623513088 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f082413e730 msgr2=0x7f082413eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:07.134 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.131+0000 7f08227fc640 1 --2- 172.21.15.150:0/1623513088 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f082413e730 0x7f082413eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.134 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.131+0000 7f08227fc640 1 -- 172.21.15.150:0/1623513088 >> v1:172.21.15.150:6789/0 conn(0x7f08240b7c10 legacy=0x7f08240a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.135 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.131+0000 7f08227fc640 10 monclient: _finish_hunting 0 2023-10-30T12:17:07.135 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.131+0000 7f08227fc640 1 monclient: found mon.noname-a 2023-10-30T12:17:07.135 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.131+0000 7f08227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:07.135 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.131+0000 7f08227fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:17:07.135 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.131+0000 7f08227fc640 1 -- 172.21.15.150:0/1623513088 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08240aa6a0 con 0x7f0824141940 2023-10-30T12:17:07.136 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.135+0000 7f08227fc640 1 --2- 172.21.15.150:0/1623513088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0824141940 0x7f0824141d90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f08180097a0 tx=0x7f081802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c944c822604ee06a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:07.145 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 1 -- 172.21.15.150:0/1623513088 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08180315a0 con 0x7f0824141940 2023-10-30T12:17:07.145 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.145 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:07.146 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 10 monclient: dump: 2023-10-30T12:17:07.146 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.146 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.146 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.146 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.146 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.147 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.147 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.147 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.147 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.147 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.148 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 10 monclient: _finish_auth 0 2023-10-30T12:17:07.148 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:17:07.148 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:07.148 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 1 -- 172.21.15.150:0/1623513088 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0818031710 con 0x7f0824141940 2023-10-30T12:17:07.148 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 5 monclient: authenticate success, global_id 4128 2023-10-30T12:17:07.149 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:07.149 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:07.149 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:07.149 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 1 -- 172.21.15.150:0/1623513088 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818031ab0 con 0x7f0824141940 2023-10-30T12:17:07.149 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.149 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 4 set_mon_vals no callback set 2023-10-30T12:17:07.150 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:07.150 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0821ffb640 10 monclient: dump: 2023-10-30T12:17:07.150 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.150 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.150 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.151 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.151 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.151 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.151 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.151 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.152 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.152 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.156 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 10 monclient: shutdown 2023-10-30T12:17:07.156 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:07.156 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 1 -- 172.21.15.150:0/1623513088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0824141940 msgr2=0x7f0824141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.156 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 1 --2- 172.21.15.150:0/1623513088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0824141940 0x7f0824141d90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f08180097a0 tx=0x7f081802b840 comp rx=0 tx=0).stop 2023-10-30T12:17:07.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 1 -- 172.21.15.150:0/1623513088 shutdown_connections 2023-10-30T12:17:07.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 1 --2- 172.21.15.150:0/1623513088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0824141940 0x7f0824141d90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 1 --2- 172.21.15.150:0/1623513088 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f082413e730 0x7f082413eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 1 -- 172.21.15.150:0/1623513088 >> 172.21.15.150:0/1623513088 conn(0x7f082406c660 msgr2=0x7f08240b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:07.157 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 1 -- 172.21.15.150:0/1623513088 shutdown_connections 2023-10-30T12:17:07.158 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.144+0000 7f0829800640 1 -- 172.21.15.150:0/1623513088 wait complete. 2023-10-30T12:17:07.158 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.145+0000 7f0829800640 10 monclient: build_initial_monmap 2023-10-30T12:17:07.158 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.145+0000 7f0829800640 10 monclient: monmap: 2023-10-30T12:17:07.158 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:07.158 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:07.159 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2023-10-30T12:17:07.159 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2023-10-30T12:17:07.159 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:07.159 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2023-10-30T12:17:07.159 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:07.159 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:07.160 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:07.160 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.160 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.145+0000 7f0829800640 1 librados: starting msgr at 2023-10-30T12:17:07.160 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.145+0000 7f0829800640 1 librados: starting objecter 2023-10-30T12:17:07.160 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 1 Processor -- start 2023-10-30T12:17:07.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 1 -- start start 2023-10-30T12:17:07.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 1 librados: setting wanted keys 2023-10-30T12:17:07.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 1 librados: calling monclient init 2023-10-30T12:17:07.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient: init 2023-10-30T12:17:07.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:07.161 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:17:07.162 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08240b7c10 0x7f08240ad900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.162 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient(hunting): picked mon.noname-a con 0x7f08240b7c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:07.162 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f082413e730 0x7f08240ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.162 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient(hunting): picked mon.noname-c con 0x7f082413e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:07.162 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0824141940 0x7f08241d7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.163 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient(hunting): picked mon.noname-b con 0x7f0824141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:07.163 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.163 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0824141ee0 con 0x7f08240b7c10 2023-10-30T12:17:07.163 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.163 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08240afad0 con 0x7f0824141940 2023-10-30T12:17:07.164 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.164 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08241d6d40 con 0x7f082413e730 2023-10-30T12:17:07.164 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:07.164 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.146+0000 7f0829800640 10 monclient(hunting): authenticate will time out at 1523.377822s 2023-10-30T12:17:07.164 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0824141940 0x7f08241d7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.164 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08227fc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f082413e730 0x7f08240ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.165 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0824141940 0x7f08241d7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38656/0 (socket says 172.21.15.150:38656) 2023-10-30T12:17:07.165 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08237fe640 1 -- 172.21.15.150:0/3033237760 learned_addr learned my addr 172.21.15.150:0/3033237760 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:07.165 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08237fe640 10 monclient(hunting): get_auth_request con 0x7f0824141940 auth_method 0 2023-10-30T12:17:07.165 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.165 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08227fc640 10 monclient(hunting): get_auth_request con 0x7f082413e730 auth_method 0 2023-10-30T12:17:07.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08227fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.147+0000 7f08227fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.166 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.148+0000 7f08237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.167 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.148+0000 7f08237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.167 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.148+0000 7f08237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.167 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.148+0000 7f08227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.167 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.148+0000 7f08227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.167 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.148+0000 7f08227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.168 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.148+0000 7f0822ffd640 1 --2- 172.21.15.150:0/3033237760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08240b7c10 0x7f08240ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.168 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.149+0000 7f0822ffd640 10 monclient(hunting): get_auth_request con 0x7f08240b7c10 auth_method 0 2023-10-30T12:17:07.168 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.149+0000 7f0822ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.168 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.149+0000 7f0822ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.168 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.149+0000 7f0822ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.168 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.149+0000 7f0822ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.169 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.149+0000 7f0822ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.169 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.149+0000 7f0822ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.169 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.151+0000 7f08237fe640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2023-10-30T12:17:07.169 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.151+0000 7f08237fe640 1 -- 172.21.15.150:0/3033237760 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f082413e730 msgr2=0x7f08240ae560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.169 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.151+0000 7f08237fe640 1 --2- 172.21.15.150:0/3033237760 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f082413e730 0x7f08240ae560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.151+0000 7f08237fe640 1 -- 172.21.15.150:0/3033237760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08240b7c10 msgr2=0x7f08240ad900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.151+0000 7f08237fe640 1 --2- 172.21.15.150:0/3033237760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08240b7c10 0x7f08240ad900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.151+0000 7f08237fe640 10 monclient: _finish_hunting 0 2023-10-30T12:17:07.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.151+0000 7f08237fe640 1 monclient: found mon.noname-b 2023-10-30T12:17:07.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.151+0000 7f08237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:07.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.151+0000 7f08237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.151+0000 7f08237fe640 1 -- 172.21.15.150:0/3033237760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08240aeaa0 con 0x7f0824141940 2023-10-30T12:17:07.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.153+0000 7f08237fe640 1 --2- 172.21.15.150:0/3033237760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0824141940 0x7f08241d7210 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f081c00efc0 tx=0x7f081c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:07.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.153+0000 7f080b7fe640 10 client.?.objecter ms_handle_connect 0x7f0824141940 2023-10-30T12:17:07.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.153+0000 7f080b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:07.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 1 -- 172.21.15.150:0/3033237760 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c00f040 con 0x7f0824141940 2023-10-30T12:17:07.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:07.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 10 monclient: dump: 2023-10-30T12:17:07.174 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.174 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.174 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.174 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.174 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.175 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.175 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.175 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.175 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.175 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.175 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:07.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:07.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:07.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 1 -- 172.21.15.150:0/3033237760 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081c014480 con 0x7f0824141940 2023-10-30T12:17:07.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:07.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 1 -- 172.21.15.150:0/3033237760 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c014840 con 0x7f0824141940 2023-10-30T12:17:07.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f0829800640 5 monclient: authenticate success, global_id 4126 2023-10-30T12:17:07.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f0829800640 10 monclient: _renew_subs 2023-10-30T12:17:07.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f0829800640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f0829800640 1 -- 172.21.15.150:0/3033237760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08240acad0 con 0x7f0824141940 2023-10-30T12:17:07.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f08217fa640 4 set_mon_vals no callback set 2023-10-30T12:17:07.178 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:07.178 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f080b7fe640 10 monclient: dump: 2023-10-30T12:17:07.178 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.178 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.178 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.179 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.179 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.179 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.179 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.179 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.180 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.180 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.180 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f0829800640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:07.180 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f0829800640 10 monclient: _renew_subs 2023-10-30T12:17:07.180 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f0829800640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.180 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f0829800640 1 -- 172.21.15.150:0/3033237760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08241d79f0 con 0x7f0824141940 2023-10-30T12:17:07.181 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.156+0000 7f0829800640 1 librados: init done 2023-10-30T12:17:07.181 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.157+0000 7f0829800640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:07.181 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.157+0000 7f0829800640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:07.181 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.157+0000 7f0829800640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.181 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.157+0000 7f0829800640 1 -- 172.21.15.150:0/3033237760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08241d6f10 con 0x7f0824141940 2023-10-30T12:17:07.182 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:07.169+0000 7fc4f022e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T12:17:07.217 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:07.216+0000 7f4e91df1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:17:07.224 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:07.223+0000 7fc4f022e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T12:17:07.259 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:07.258+0000 7f4e91df1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T12:17:07.263 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.261+0000 7f080b7fe640 1 -- 172.21.15.150:0/3033237760 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f081c014cb0 con 0x7f0824141940 2023-10-30T12:17:07.263 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.261+0000 7f080b7fe640 1 -- 172.21.15.150:0/3033237760 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f081c026d50 con 0x7f0824141940 2023-10-30T12:17:07.263 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.261+0000 7f080b7fe640 10 client.4126.objecter ms_dispatch 0x7f0824000b90 osd_map(5..5 src has 1..5) v4 2023-10-30T12:17:07.264 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.261+0000 7f080b7fe640 3 client.4126.objecter handle_osd_map got epochs [5,5] > 0 2023-10-30T12:17:07.264 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.261+0000 7f080b7fe640 3 client.4126.objecter handle_osd_map decoding full epoch 5 2023-10-30T12:17:07.264 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.262+0000 7f080b7fe640 20 client.4126.objecter dump_active .. 0 homeless 2023-10-30T12:17:07.264 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.262+0000 7f080b7fe640 1 -- 172.21.15.150:0/3033237760 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f081c0138d0 con 0x7f0824141940 2023-10-30T12:17:07.264 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.262+0000 7f080b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:07.265 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.262+0000 7f080b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:07.270 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:07.269+0000 7fc4f022e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T12:17:07.325 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:07.324+0000 7fc4f022e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T12:17:07.334 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.333+0000 7f0829800640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "36c6a615-3a6a-4c78-86ed-546ddf04aa1f", "id": 3}] 2023-10-30T12:17:07.335 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.333+0000 7f0829800640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "36c6a615-3a6a-4c78-86ed-546ddf04aa1f", "id": 3}] 2023-10-30T12:17:07.335 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.333+0000 7f0829800640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.335 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.333+0000 7f0829800640 1 -- 172.21.15.150:0/3033237760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "36c6a615-3a6a-4c78-86ed-546ddf04aa1f", "id": 3} v 0) v1 -- 0x7f0824075550 con 0x7f0824141940 2023-10-30T12:17:07.369 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:07.367+0000 7fc4f022e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T12:17:07.406 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:07.404+0000 7f4e91df1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T12:17:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.405+0000 7f080b7fe640 1 -- 172.21.15.150:0/3033237760 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "36c6a615-3a6a-4c78-86ed-546ddf04aa1f", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f081c02a020 con 0x7f0824141940 2023-10-30T12:17:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.405+0000 7f080b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "36c6a615-3a6a-4c78-86ed-546ddf04aa1f", "id": 3}] 2023-10-30T12:17:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.405+0000 7f080b7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:07.408 INFO:teuthology.orchestra.run.smithi150.stdout:3 2023-10-30T12:17:07.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 10 librados: watch_flush enter 2023-10-30T12:17:07.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 10 librados: watch_flush exit 2023-10-30T12:17:07.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 20 client.4126.objecter shutdown clearing up homeless session... 2023-10-30T12:17:07.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 10 client.4126.objecter successfully canceled tick 2023-10-30T12:17:07.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 10 monclient: shutdown 2023-10-30T12:17:07.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:07.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 1 -- 172.21.15.150:0/3033237760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0824141940 msgr2=0x7f08241d7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 1 --2- 172.21.15.150:0/3033237760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0824141940 0x7f08241d7210 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f081c00efc0 tx=0x7f081c00c490 comp rx=0 tx=0).stop 2023-10-30T12:17:07.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 1 -- 172.21.15.150:0/3033237760 shutdown_connections 2023-10-30T12:17:07.411 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 1 --2- 172.21.15.150:0/3033237760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0824141940 0x7f08241d7210 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.411 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 1 --2- 172.21.15.150:0/3033237760 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f082413e730 0x7f08240ae560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.411 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 1 --2- 172.21.15.150:0/3033237760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08240b7c10 0x7f08240ad900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.411 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 1 -- 172.21.15.150:0/3033237760 >> 172.21.15.150:0/3033237760 conn(0x7f082406c660 msgr2=0x7f08240a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:07.411 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 1 -- 172.21.15.150:0/3033237760 shutdown_connections 2023-10-30T12:17:07.412 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 1 -- 172.21.15.150:0/3033237760 wait complete. 2023-10-30T12:17:07.412 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.407+0000 7f0829800640 1 librados: shutdown 2023-10-30T12:17:07.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 2cb5add2-1200-41f1-8e70-78abdc2df024 4 2023-10-30T12:17:07.476 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:07.475+0000 7fc4f022e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-30T12:17:07.518 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.516+0000 7f3b307f5640 10 monclient: get_monmap_and_config 2023-10-30T12:17:07.518 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.516+0000 7f3b307f5640 10 monclient: build_initial_monmap 2023-10-30T12:17:07.518 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.516+0000 7f3b307f5640 10 monclient: monmap: 2023-10-30T12:17:07.519 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:07.519 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.519 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:17:07.517491+0000 2023-10-30T12:17:07.519 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:17:07.517491+0000 2023-10-30T12:17:07.519 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:07.520 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.520 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:07.520 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:07.520 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:07.520 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:07.521 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:07.521 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.521 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 1 Processor -- start 2023-10-30T12:17:07.521 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 1 -- start start 2023-10-30T12:17:07.521 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient: init 2023-10-30T12:17:07.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:07.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T12:17:07.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3b280b7c10 0x7f3b280a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient(hunting): picked mon.noname-c con 0x7f3b280b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:07.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3b28141940 0x7f3b28141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient(hunting): picked mon.noname-a con 0x7f3b28141940 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:07.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3b280b1c10 0x7f3b280b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient(hunting): picked mon.noname-b con 0x7f3b280b1c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:07.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b280670a0 con 0x7f3b28141940 2023-10-30T12:17:07.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b280b7ff0 con 0x7f3b280b7c10 2023-10-30T12:17:07.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.527 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b280a9350 con 0x7f3b280b1c10 2023-10-30T12:17:07.527 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:07.527 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.517+0000 7f3b307f5640 10 monclient(hunting): authenticate will time out at 1523.748895s 2023-10-30T12:17:07.527 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2e591640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3b280b7c10 0x7f3b280a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.527 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2ed92640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3b280b1c10 0x7f3b280b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.528 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2dd90640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3b28141940 0x7f3b28141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.528 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2e591640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3b280b7c10 0x7f3b280a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38664/0 (socket says 172.21.15.150:38664) 2023-10-30T12:17:07.528 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2e591640 1 -- 172.21.15.150:0/3956993040 learned_addr learned my addr 172.21.15.150:0/3956993040 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:07.528 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2e591640 10 monclient(hunting): get_auth_request con 0x7f3b280b7c10 auth_method 0 2023-10-30T12:17:07.528 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2e591640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2e591640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2e591640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2ed92640 10 monclient(hunting): get_auth_request con 0x7f3b280b1c10 auth_method 0 2023-10-30T12:17:07.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2ed92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.518+0000 7f3b2ed92640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.530 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2ed92640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.530 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2dd90640 10 monclient(hunting): get_auth_request con 0x7f3b28141940 auth_method 0 2023-10-30T12:17:07.530 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2dd90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.530 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2dd90640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.530 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2dd90640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.530 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2e591640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2e591640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2e591640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2ed92640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2ed92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2ed92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.532 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2dd90640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.532 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2dd90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.532 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.519+0000 7f3b2dd90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.532 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.520+0000 7f3b2e591640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2023-10-30T12:17:07.532 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.520+0000 7f3b2e591640 1 -- 172.21.15.150:0/3956993040 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3b280b1c10 msgr2=0x7f3b280b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:07.533 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.521+0000 7f3b2e591640 1 --2- 172.21.15.150:0/3956993040 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3b280b1c10 0x7f3b280b2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.533 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.521+0000 7f3b2e591640 1 -- 172.21.15.150:0/3956993040 >> v2:172.21.15.148:3300/0 conn(0x7f3b28141940 msgr2=0x7f3b28141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:07.533 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.521+0000 7f3b2e591640 1 --2- 172.21.15.150:0/3956993040 >> v2:172.21.15.148:3300/0 conn(0x7f3b28141940 0x7f3b28141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.533 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.521+0000 7f3b2e591640 10 monclient: _finish_hunting 0 2023-10-30T12:17:07.533 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.521+0000 7f3b2e591640 1 monclient: found mon.noname-c 2023-10-30T12:17:07.533 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.521+0000 7f3b2e591640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:07.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.521+0000 7f3b2e591640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.521+0000 7f3b2e591640 1 -- 172.21.15.150:0/3956993040 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b280aa6a0 con 0x7f3b280b7c10 2023-10-30T12:17:07.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.522+0000 7f3b2e591640 1 --2- 172.21.15.150:0/3956993040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 0x7f3b280a8aa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b280b4d80 tx=0x7f3b2402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59ad8366b667135c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:07.538 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.525+0000 7f3b2d58f640 1 -- 172.21.15.150:0/3956993040 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b240317e0 con 0x7f3b280b7c10 2023-10-30T12:17:07.538 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.538 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:07.538 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 10 monclient: dump: 2023-10-30T12:17:07.538 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.539 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.539 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.539 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.539 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.539 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.540 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.540 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.540 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.540 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.540 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 10 monclient: _finish_auth 0 2023-10-30T12:17:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 10 monclient: _check_auth_tickets 2023-10-30T12:17:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 1 -- 172.21.15.150:0/3956993040 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b24031950 con 0x7f3b280b7c10 2023-10-30T12:17:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 1 -- 172.21.15.150:0/3956993040 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24031cf0 con 0x7f3b280b7c10 2023-10-30T12:17:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 5 monclient: authenticate success, global_id 4129 2023-10-30T12:17:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 4 set_mon_vals no callback set 2023-10-30T12:17:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b2d58f640 10 monclient: dump: 2023-10-30T12:17:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.544 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.544 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.544 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.544 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.544 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.544 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 10 monclient: shutdown 2023-10-30T12:17:07.545 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:07.545 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 1 -- 172.21.15.150:0/3956993040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 msgr2=0x7f3b280a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.545 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 1 --2- 172.21.15.150:0/3956993040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 0x7f3b280a8aa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b280b4d80 tx=0x7f3b2402b880 comp rx=0 tx=0).stop 2023-10-30T12:17:07.545 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 1 -- 172.21.15.150:0/3956993040 shutdown_connections 2023-10-30T12:17:07.545 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 1 --2- 172.21.15.150:0/3956993040 >> v2:172.21.15.148:3300/0 conn(0x7f3b28141940 0x7f3b28141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 1 --2- 172.21.15.150:0/3956993040 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3b280b1c10 0x7f3b280b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 1 --2- 172.21.15.150:0/3956993040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 0x7f3b280a8aa0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 1 -- 172.21.15.150:0/3956993040 >> 172.21.15.150:0/3956993040 conn(0x7f3b2806c660 msgr2=0x7f3b280b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:07.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 1 -- 172.21.15.150:0/3956993040 shutdown_connections 2023-10-30T12:17:07.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.526+0000 7f3b307f5640 1 -- 172.21.15.150:0/3956993040 wait complete. 2023-10-30T12:17:07.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.527+0000 7f3b307f5640 10 monclient: build_initial_monmap 2023-10-30T12:17:07.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.527+0000 7f3b307f5640 10 monclient: monmap: 2023-10-30T12:17:07.547 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:07.547 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:07.547 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2023-10-30T12:17:07.547 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2023-10-30T12:17:07.548 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:07.548 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2023-10-30T12:17:07.548 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:07.548 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:07.548 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:07.549 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 librados: starting msgr at 2023-10-30T12:17:07.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 librados: starting objecter 2023-10-30T12:17:07.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 Processor -- start 2023-10-30T12:17:07.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 -- start start 2023-10-30T12:17:07.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 librados: setting wanted keys 2023-10-30T12:17:07.550 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 librados: calling monclient init 2023-10-30T12:17:07.550 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient: init 2023-10-30T12:17:07.550 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:07.550 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:17:07.550 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b280b1c10 0x7f3b281c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.551 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient(hunting): picked mon.noname-a con 0x7f3b280b1c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:07.551 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 0x7f3b280b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.551 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient(hunting): picked mon.noname-b con 0x7f3b280b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:07.551 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3b28141940 0x7f3b281cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.551 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient(hunting): picked mon.noname-c con 0x7f3b28141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:07.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b280b16d0 con 0x7f3b280b1c10 2023-10-30T12:17:07.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b281c8460 con 0x7f3b280b7c10 2023-10-30T12:17:07.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b281c8650 con 0x7f3b28141940 2023-10-30T12:17:07.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:07.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.528+0000 7f3b307f5640 10 monclient(hunting): authenticate will time out at 1523.759702s 2023-10-30T12:17:07.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2dd90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 0x7f3b280b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2e591640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b280b1c10 0x7f3b281c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2ed92640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3b28141940 0x7f3b281cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2dd90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 0x7f3b280b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38676/0 (socket says 172.21.15.150:38676) 2023-10-30T12:17:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2dd90640 1 -- 172.21.15.150:0/1387034009 learned_addr learned my addr 172.21.15.150:0/1387034009 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2dd90640 10 monclient(hunting): get_auth_request con 0x7f3b280b7c10 auth_method 0 2023-10-30T12:17:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2dd90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2dd90640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2dd90640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2ed92640 10 monclient(hunting): get_auth_request con 0x7f3b28141940 auth_method 0 2023-10-30T12:17:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2ed92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2ed92640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2ed92640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.556 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2e591640 10 monclient(hunting): get_auth_request con 0x7f3b280b1c10 auth_method 0 2023-10-30T12:17:07.556 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2e591640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.556 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2e591640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.556 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.529+0000 7f3b2e591640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.556 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.530+0000 7f3b2dd90640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.530+0000 7f3b2dd90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.530+0000 7f3b2dd90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.530+0000 7f3b2ed92640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.530+0000 7f3b2ed92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.530+0000 7f3b2ed92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.530+0000 7f3b2e591640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.530+0000 7f3b2e591640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.530+0000 7f3b2e591640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.532+0000 7f3b2dd90640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2023-10-30T12:17:07.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.532+0000 7f3b2dd90640 1 -- 172.21.15.150:0/1387034009 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3b28141940 msgr2=0x7f3b281cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.532+0000 7f3b2dd90640 1 --2- 172.21.15.150:0/1387034009 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3b28141940 0x7f3b281cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.559 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.532+0000 7f3b2dd90640 1 -- 172.21.15.150:0/1387034009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b280b1c10 msgr2=0x7f3b281c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.559 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.532+0000 7f3b2dd90640 1 --2- 172.21.15.150:0/1387034009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b280b1c10 0x7f3b281c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.559 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.532+0000 7f3b2dd90640 10 monclient: _finish_hunting 0 2023-10-30T12:17:07.559 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.532+0000 7f3b2dd90640 1 monclient: found mon.noname-b 2023-10-30T12:17:07.559 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.532+0000 7f3b2dd90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:07.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.532+0000 7f3b2dd90640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.532+0000 7f3b2dd90640 1 -- 172.21.15.150:0/1387034009 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b280b10d0 con 0x7f3b280b7c10 2023-10-30T12:17:07.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.534+0000 7f3b2dd90640 1 --2- 172.21.15.150:0/1387034009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 0x7f3b280b0b90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800e8a0 tx=0x7f3b1800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:07.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.534+0000 7f3b1effd640 10 client.?.objecter ms_handle_connect 0x7f3b280b7c10 2023-10-30T12:17:07.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.534+0000 7f3b1effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:07.562 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 1 -- 172.21.15.150:0/1387034009 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1800c940 con 0x7f3b280b7c10 2023-10-30T12:17:07.562 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.562 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:07.562 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 10 monclient: dump: 2023-10-30T12:17:07.563 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.563 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.563 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.563 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.563 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.564 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.564 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.564 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.564 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.564 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.565 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 10 monclient: _finish_auth 0 2023-10-30T12:17:07.565 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:07.565 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:07.565 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 1 -- 172.21.15.150:0/1387034009 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1800d040 con 0x7f3b280b7c10 2023-10-30T12:17:07.565 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:07.566 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 1 -- 172.21.15.150:0/1387034009 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1800ccf0 con 0x7f3b280b7c10 2023-10-30T12:17:07.566 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.566 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:07.566 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b1effd640 10 monclient: dump: 2023-10-30T12:17:07.566 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.567 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.567 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.567 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.567 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.567 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.568 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.568 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.568 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.568 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.568 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b307f5640 5 monclient: authenticate success, global_id 4132 2023-10-30T12:17:07.568 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b307f5640 10 monclient: _renew_subs 2023-10-30T12:17:07.569 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b307f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.569 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b2cd8e640 4 set_mon_vals no callback set 2023-10-30T12:17:07.569 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b307f5640 1 -- 172.21.15.150:0/1387034009 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b280afce0 con 0x7f3b280b7c10 2023-10-30T12:17:07.569 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b307f5640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:07.569 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b307f5640 10 monclient: _renew_subs 2023-10-30T12:17:07.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b307f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b307f5640 1 -- 172.21.15.150:0/1387034009 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b281ce540 con 0x7f3b280b7c10 2023-10-30T12:17:07.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.538+0000 7f3b307f5640 1 librados: init done 2023-10-30T12:17:07.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.539+0000 7f3b307f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:07.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.539+0000 7f3b307f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.539+0000 7f3b307f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.539+0000 7f3b307f5640 1 -- 172.21.15.150:0/1387034009 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2813a220 con 0x7f3b280b7c10 2023-10-30T12:17:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.541+0000 7f3b1effd640 1 -- 172.21.15.150:0/1387034009 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3b18005c90 con 0x7f3b280b7c10 2023-10-30T12:17:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.541+0000 7f3b1effd640 1 -- 172.21.15.150:0/1387034009 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f3b18006690 con 0x7f3b280b7c10 2023-10-30T12:17:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.541+0000 7f3b1effd640 10 client.4132.objecter ms_dispatch 0x7f3b28000b90 osd_map(6..6 src has 1..6) v4 2023-10-30T12:17:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.541+0000 7f3b1effd640 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2023-10-30T12:17:07.572 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.541+0000 7f3b1effd640 3 client.4132.objecter handle_osd_map decoding full epoch 6 2023-10-30T12:17:07.572 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.541+0000 7f3b1effd640 20 client.4132.objecter dump_active .. 0 homeless 2023-10-30T12:17:07.572 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:07.549+0000 7fc4f022e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T12:17:07.598 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:07.596+0000 7fc4f022e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T12:17:07.642 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.640+0000 7f3b1effd640 1 -- 172.21.15.150:0/1387034009 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f3b18006860 con 0x7f3b280b7c10 2023-10-30T12:17:07.642 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.640+0000 7f3b1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:07.642 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.640+0000 7f3b1effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:07.651 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:07.650+0000 7fc4f022e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T12:17:07.699 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:07.698+0000 7fc4f022e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T12:17:07.713 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.712+0000 7f3b307f5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2cb5add2-1200-41f1-8e70-78abdc2df024", "id": 4}] 2023-10-30T12:17:07.714 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.712+0000 7f3b307f5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2cb5add2-1200-41f1-8e70-78abdc2df024", "id": 4}] 2023-10-30T12:17:07.714 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.712+0000 7f3b307f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:07.714 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.712+0000 7f3b307f5640 1 -- 172.21.15.150:0/1387034009 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2cb5add2-1200-41f1-8e70-78abdc2df024", "id": 4} v 0) v1 -- 0x7f3b280a97c0 con 0x7f3b280b7c10 2023-10-30T12:17:07.782 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.780+0000 7f3b1effd640 1 -- 172.21.15.150:0/1387034009 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2cb5add2-1200-41f1-8e70-78abdc2df024", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3b1800ce60 con 0x7f3b280b7c10 2023-10-30T12:17:07.782 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.780+0000 7f3b1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2cb5add2-1200-41f1-8e70-78abdc2df024", "id": 4}] 2023-10-30T12:17:07.782 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.780+0000 7f3b1effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:07.783 INFO:teuthology.orchestra.run.smithi150.stdout:4 2023-10-30T12:17:07.784 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 10 librados: watch_flush enter 2023-10-30T12:17:07.784 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 10 librados: watch_flush exit 2023-10-30T12:17:07.784 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 20 client.4132.objecter shutdown clearing up homeless session... 2023-10-30T12:17:07.785 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 10 client.4132.objecter successfully canceled tick 2023-10-30T12:17:07.785 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 10 monclient: shutdown 2023-10-30T12:17:07.785 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:07.785 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 1 -- 172.21.15.150:0/1387034009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 msgr2=0x7f3b280b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.785 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 1 --2- 172.21.15.150:0/1387034009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 0x7f3b280b0b90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800e8a0 tx=0x7f3b1800ed60 comp rx=0 tx=0).stop 2023-10-30T12:17:07.785 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 1 -- 172.21.15.150:0/1387034009 shutdown_connections 2023-10-30T12:17:07.786 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 1 --2- 172.21.15.150:0/1387034009 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3b28141940 0x7f3b281cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.786 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 1 --2- 172.21.15.150:0/1387034009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b280b7c10 0x7f3b280b0b90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.786 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 1 --2- 172.21.15.150:0/1387034009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b280b1c10 0x7f3b281c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.786 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.781+0000 7f3b307f5640 1 -- 172.21.15.150:0/1387034009 >> 172.21.15.150:0/1387034009 conn(0x7f3b2806c660 msgr2=0x7f3b2813db20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:07.786 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.782+0000 7f3b307f5640 1 -- 172.21.15.150:0/1387034009 shutdown_connections 2023-10-30T12:17:07.787 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.782+0000 7f3b307f5640 1 -- 172.21.15.150:0/1387034009 wait complete. 2023-10-30T12:17:07.787 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.782+0000 7f3b307f5640 1 librados: shutdown 2023-10-30T12:17:07.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb 5 2023-10-30T12:17:07.892 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.891+0000 7f6154ff0640 10 monclient: get_monmap_and_config 2023-10-30T12:17:07.893 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.891+0000 7f6154ff0640 10 monclient: build_initial_monmap 2023-10-30T12:17:07.893 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.891+0000 7f6154ff0640 10 monclient: monmap: 2023-10-30T12:17:07.893 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:07.893 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.894 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:17:07.892205+0000 2023-10-30T12:17:07.894 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:17:07.892205+0000 2023-10-30T12:17:07.894 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:07.894 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.894 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:07.895 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:07.895 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:07.895 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:07.895 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:07.895 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.896 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.891+0000 7f6154ff0640 1 Processor -- start 2023-10-30T12:17:07.896 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.891+0000 7f6154ff0640 1 -- start start 2023-10-30T12:17:07.896 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.891+0000 7f6154ff0640 10 monclient: init 2023-10-30T12:17:07.898 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:07.899 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-10-30T12:17:07.899 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f61500b7c10 0x7f61500a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.899 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 10 monclient(hunting): picked mon.noname-a con 0x7f61500b7c10 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:07.899 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6150141940 0x7f6150141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.899 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 10 monclient(hunting): picked mon.noname-c con 0x7f6150141940 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:07.900 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61500b1c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:07.900 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61500a9350 con 0x7f61500b1c10 2023-10-30T12:17:07.900 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.900 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61500670a0 con 0x7f61500b7c10 2023-10-30T12:17:07.900 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.901 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61500b7ff0 con 0x7f6150141940 2023-10-30T12:17:07.901 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:07.901 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.892+0000 7f6154ff0640 10 monclient(hunting): authenticate will time out at 1524.123631s 2023-10-30T12:17:07.901 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.893+0000 7f614dd9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6150141940 0x7f6150141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.901 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.893+0000 7f614e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f61500b7c10 0x7f61500a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.901 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.893+0000 7f614e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f61500b7c10 0x7f61500a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.150:43730/0 (socket says 172.21.15.150:43730) 2023-10-30T12:17:07.902 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.893+0000 7f614e59c640 1 -- 172.21.15.150:0/327473992 learned_addr learned my addr 172.21.15.150:0/327473992 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:07.902 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.893+0000 7f614e59c640 10 monclient(hunting): get_auth_request con 0x7f61500b7c10 auth_method 0 2023-10-30T12:17:07.902 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.893+0000 7f614e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.902 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.893+0000 7f614e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.902 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.893+0000 7f614e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.894+0000 7f614dd9b640 10 monclient(hunting): get_auth_request con 0x7f6150141940 auth_method 0 2023-10-30T12:17:07.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.894+0000 7f614dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.894+0000 7f614dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.894+0000 7f614dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.894+0000 7f614e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.904 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.894+0000 7f614e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.904 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.894+0000 7f614e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.904 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.904 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.904 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614e59c640 10 monclient(hunting): handle_auth_done global_id 4140 payload 274 2023-10-30T12:17:07.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614e59c640 1 -- 172.21.15.150:0/327473992 >> v2:172.21.15.150:3300/0 conn(0x7f6150141940 msgr2=0x7f6150141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:07.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614e59c640 1 --2- 172.21.15.150:0/327473992 >> v2:172.21.15.150:3300/0 conn(0x7f6150141940 0x7f6150141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614e59c640 1 -- 172.21.15.150:0/327473992 >> v1:172.21.15.150:6789/0 conn(0x7f61500b1c10 legacy=0x7f61500b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:17:07.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614e59c640 1 monclient: found mon.noname-a 2023-10-30T12:17:07.906 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:07.906 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:17:07.906 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.895+0000 7f614e59c640 1 -- 172.21.15.150:0/327473992 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61500aa6a0 con 0x7f61500b7c10 2023-10-30T12:17:07.906 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.897+0000 7f614e59c640 1 --2- 172.21.15.150:0/327473992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61500b7c10 0x7f61500a8aa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61500a9c40 tx=0x7f614002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352c92bdadfa5ec4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:07.910 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 1 -- 172.21.15.150:0/327473992 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140031870 con 0x7f61500b7c10 2023-10-30T12:17:07.910 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.911 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:07.911 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 10 monclient: dump: 2023-10-30T12:17:07.911 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.911 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.911 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.912 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.912 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.912 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.912 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.912 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.913 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.913 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.913 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 10 monclient: _finish_auth 0 2023-10-30T12:17:07.913 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:17:07.913 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:07.914 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 1 -- 172.21.15.150:0/327473992 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61400319e0 con 0x7f61500b7c10 2023-10-30T12:17:07.914 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f6154ff0640 5 monclient: authenticate success, global_id 4140 2023-10-30T12:17:07.914 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f6154ff0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:07.914 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:07.914 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f6154ff0640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:07.915 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f6154ff0640 4 set_mon_vals no callback set 2023-10-30T12:17:07.915 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 1 -- 172.21.15.150:0/327473992 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140031d80 con 0x7f61500b7c10 2023-10-30T12:17:07.915 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.915 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:07.915 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.901+0000 7f614d59a640 10 monclient: dump: 2023-10-30T12:17:07.915 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.916 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.916 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.916 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.916 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.916 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.917 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.917 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.917 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.917 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.917 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.902+0000 7f6154ff0640 10 monclient: shutdown 2023-10-30T12:17:07.918 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.902+0000 7f6154ff0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:07.918 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.902+0000 7f6154ff0640 1 -- 172.21.15.150:0/327473992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61500b7c10 msgr2=0x7f61500a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.918 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.902+0000 7f6154ff0640 1 --2- 172.21.15.150:0/327473992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61500b7c10 0x7f61500a8aa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61500a9c40 tx=0x7f614002b910 comp rx=0 tx=0).stop 2023-10-30T12:17:07.918 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.902+0000 7f6154ff0640 1 -- 172.21.15.150:0/327473992 shutdown_connections 2023-10-30T12:17:07.918 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.902+0000 7f6154ff0640 1 --2- 172.21.15.150:0/327473992 >> v2:172.21.15.150:3300/0 conn(0x7f6150141940 0x7f6150141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.918 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.902+0000 7f6154ff0640 1 --2- 172.21.15.150:0/327473992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61500b7c10 0x7f61500a8aa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.919 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.902+0000 7f6154ff0640 1 -- 172.21.15.150:0/327473992 >> 172.21.15.150:0/327473992 conn(0x7f615006c660 msgr2=0x7f61500b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:07.919 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.902+0000 7f6154ff0640 1 -- 172.21.15.150:0/327473992 shutdown_connections 2023-10-30T12:17:07.919 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.902+0000 7f6154ff0640 1 -- 172.21.15.150:0/327473992 wait complete. 2023-10-30T12:17:07.919 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.903+0000 7f6154ff0640 10 monclient: build_initial_monmap 2023-10-30T12:17:07.919 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.903+0000 7f6154ff0640 10 monclient: monmap: 2023-10-30T12:17:07.920 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:07.920 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:07.920 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2023-10-30T12:17:07.920 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2023-10-30T12:17:07.920 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:07.921 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2023-10-30T12:17:07.921 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:07.921 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:07.921 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:07.921 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.921 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.903+0000 7f6154ff0640 1 librados: starting msgr at 2023-10-30T12:17:07.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.903+0000 7f6154ff0640 1 librados: starting objecter 2023-10-30T12:17:07.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.904+0000 7f6154ff0640 1 Processor -- start 2023-10-30T12:17:07.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.904+0000 7f6154ff0640 1 -- start start 2023-10-30T12:17:07.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.904+0000 7f6154ff0640 1 librados: setting wanted keys 2023-10-30T12:17:07.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.904+0000 7f6154ff0640 1 librados: calling monclient init 2023-10-30T12:17:07.923 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.904+0000 7f6154ff0640 10 monclient: init 2023-10-30T12:17:07.923 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:07.923 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:17:07.923 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f61500b1c10 0x7f61500ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.923 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 10 monclient(hunting): picked mon.noname-c con 0x7f61500b1c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:07.924 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61500b7c10 0x7f61500af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.924 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 10 monclient(hunting): picked mon.noname-a con 0x7f61500b7c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:07.924 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6150141940 0x7f61501d6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:07.924 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 10 monclient(hunting): picked mon.noname-b con 0x7f6150141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:07.924 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61500a8640 con 0x7f61500b7c10 2023-10-30T12:17:07.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61500b0be0 con 0x7f6150141940 2023-10-30T12:17:07.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:07.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61501d66d0 con 0x7f61500b1c10 2023-10-30T12:17:07.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:07.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f6154ff0640 10 monclient(hunting): authenticate will time out at 1524.136063s 2023-10-30T12:17:07.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6150141940 0x7f61501d6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614dd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61500b7c10 0x7f61500af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614e59c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f61500b1c10 0x7f61500ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:07.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6150141940 0x7f61501d6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38692/0 (socket says 172.21.15.150:38692) 2023-10-30T12:17:07.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614ed9d640 1 -- 172.21.15.150:0/3082081066 learned_addr learned my addr 172.21.15.150:0/3082081066 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:07.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614ed9d640 10 monclient(hunting): get_auth_request con 0x7f6150141940 auth_method 0 2023-10-30T12:17:07.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614dd9b640 10 monclient(hunting): get_auth_request con 0x7f61500b7c10 auth_method 0 2023-10-30T12:17:07.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.905+0000 7f614dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614e59c640 10 monclient(hunting): get_auth_request con 0x7f61500b1c10 auth_method 0 2023-10-30T12:17:07.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:07.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:07.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:07.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.930 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.930 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.930 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.930 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:07.930 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:07.931 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.906+0000 7f614e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:07.931 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.908+0000 7f614dd9b640 10 monclient(hunting): handle_auth_done global_id 4143 payload 824 2023-10-30T12:17:07.931 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.908+0000 7f614dd9b640 1 -- 172.21.15.150:0/3082081066 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f61500b1c10 msgr2=0x7f61500ae780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.931 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.908+0000 7f614dd9b640 1 --2- 172.21.15.150:0/3082081066 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f61500b1c10 0x7f61500ae780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.931 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.908+0000 7f614dd9b640 1 -- 172.21.15.150:0/3082081066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6150141940 msgr2=0x7f61501d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:07.932 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.908+0000 7f614dd9b640 1 --2- 172.21.15.150:0/3082081066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6150141940 0x7f61501d6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:07.932 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.908+0000 7f614dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:17:07.932 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.908+0000 7f614dd9b640 1 monclient: found mon.noname-a 2023-10-30T12:17:07.932 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.908+0000 7f614dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:07.932 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.908+0000 7f614dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:17:07.932 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.908+0000 7f614dd9b640 1 -- 172.21.15.150:0/3082081066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61500af810 con 0x7f61500b7c10 2023-10-30T12:17:07.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.911+0000 7f614dd9b640 1 --2- 172.21.15.150:0/3082081066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61500b7c10 0x7f61500af2d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f614400e870 tx=0x7f614400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:07.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.911+0000 7f613effd640 10 client.?.objecter ms_handle_connect 0x7f61500b7c10 2023-10-30T12:17:07.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.911+0000 7f613effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:07.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f613effd640 1 -- 172.21.15.150:0/3082081066 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614400ee60 con 0x7f61500b7c10 2023-10-30T12:17:07.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f613effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f613effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:07.936 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f613effd640 10 monclient: dump: 2023-10-30T12:17:07.936 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.936 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.936 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.936 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.937 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.937 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.937 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.937 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.937 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.938 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.938 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f613effd640 10 monclient: _finish_auth 0 2023-10-30T12:17:07.938 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f613effd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:07.938 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f613effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:07.938 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f6154ff0640 5 monclient: authenticate success, global_id 4143 2023-10-30T12:17:07.938 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f6154ff0640 10 monclient: _renew_subs 2023-10-30T12:17:07.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f6154ff0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:07.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f6154ff0640 1 -- 172.21.15.150:0/3082081066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61500ad8d0 con 0x7f61500b7c10 2023-10-30T12:17:07.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f6154ff0640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:07.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f6154ff0640 10 monclient: _renew_subs 2023-10-30T12:17:07.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f6154ff0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:07.940 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f6154ff0640 1 -- 172.21.15.150:0/3082081066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61501d70f0 con 0x7f61500b7c10 2023-10-30T12:17:07.940 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.913+0000 7f6154ff0640 1 librados: init done 2023-10-30T12:17:07.940 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f6154ff0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:07.940 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f6154ff0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:07.940 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f6154ff0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:07.941 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f6154ff0640 1 -- 172.21.15.150:0/3082081066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61500a97c0 con 0x7f61500b7c10 2023-10-30T12:17:07.941 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f613effd640 1 -- 172.21.15.150:0/3082081066 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614400d040 con 0x7f61500b7c10 2023-10-30T12:17:07.941 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f613effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:07.941 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f613effd640 1 -- 172.21.15.150:0/3082081066 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6144005b90 con 0x7f61500b7c10 2023-10-30T12:17:07.941 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f613effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:07.941 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f613effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:07.942 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f613effd640 10 monclient: dump: 2023-10-30T12:17:07.942 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:07.942 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:07.942 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.942 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:07.943 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:07.943 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:07.943 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:07.943 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:07.943 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:07.944 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:07.944 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:07.914+0000 7f614cd99640 4 set_mon_vals no callback set 2023-10-30T12:17:08.031 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.029+0000 7f613effd640 1 -- 172.21.15.150:0/3082081066 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f614400cc60 con 0x7f61500b7c10 2023-10-30T12:17:08.031 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.030+0000 7f613effd640 1 -- 172.21.15.150:0/3082081066 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f61440065d0 con 0x7f61500b7c10 2023-10-30T12:17:08.031 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.030+0000 7f613effd640 10 client.4143.objecter ms_dispatch 0x7f6150000b90 osd_map(7..7 src has 1..7) v4 2023-10-30T12:17:08.032 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.030+0000 7f613effd640 3 client.4143.objecter handle_osd_map got epochs [7,7] > 0 2023-10-30T12:17:08.032 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.030+0000 7f613effd640 3 client.4143.objecter handle_osd_map decoding full epoch 7 2023-10-30T12:17:08.032 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.030+0000 7f613effd640 20 client.4143.objecter dump_active .. 0 homeless 2023-10-30T12:17:08.035 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.034+0000 7f613effd640 1 -- 172.21.15.150:0/3082081066 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f61440067c0 con 0x7f61500b7c10 2023-10-30T12:17:08.036 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.034+0000 7f613effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:08.036 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.034+0000 7f613effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:08.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:08.091+0000 7fc4f022e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-30T12:17:08.107 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.105+0000 7f6154ff0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb", "id": 5}] 2023-10-30T12:17:08.107 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.105+0000 7f6154ff0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb", "id": 5}] 2023-10-30T12:17:08.107 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.105+0000 7f6154ff0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:08.107 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.105+0000 7f6154ff0640 1 -- 172.21.15.150:0/3082081066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb", "id": 5} v 0) v1 -- 0x7f61500a8fb0 con 0x7f61500b7c10 2023-10-30T12:17:08.222 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.221+0000 7f613effd640 1 -- 172.21.15.150:0/3082081066 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6144005d00 con 0x7f61500b7c10 2023-10-30T12:17:08.223 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.221+0000 7f613effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb", "id": 5}] 2023-10-30T12:17:08.223 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.221+0000 7f613effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:08.224 INFO:teuthology.orchestra.run.smithi150.stdout:5 2023-10-30T12:17:08.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 10 librados: watch_flush enter 2023-10-30T12:17:08.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 10 librados: watch_flush exit 2023-10-30T12:17:08.226 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 20 client.4143.objecter shutdown clearing up homeless session... 2023-10-30T12:17:08.226 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 10 client.4143.objecter successfully canceled tick 2023-10-30T12:17:08.226 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 10 monclient: shutdown 2023-10-30T12:17:08.226 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:08.226 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 1 -- 172.21.15.150:0/3082081066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61500b7c10 msgr2=0x7f61500af2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 1 --2- 172.21.15.150:0/3082081066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61500b7c10 0x7f61500af2d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f614400e870 tx=0x7f614400ed30 comp rx=0 tx=0).stop 2023-10-30T12:17:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 1 -- 172.21.15.150:0/3082081066 shutdown_connections 2023-10-30T12:17:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 1 --2- 172.21.15.150:0/3082081066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6150141940 0x7f61501d6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 1 --2- 172.21.15.150:0/3082081066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61500b7c10 0x7f61500af2d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 1 --2- 172.21.15.150:0/3082081066 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f61500b1c10 0x7f61500ae780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 1 -- 172.21.15.150:0/3082081066 >> 172.21.15.150:0/3082081066 conn(0x7f615006c660 msgr2=0x7f61500b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 1 -- 172.21.15.150:0/3082081066 shutdown_connections 2023-10-30T12:17:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 1 -- 172.21.15.150:0/3082081066 wait complete. 2023-10-30T12:17:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.222+0000 7f6154ff0640 1 librados: shutdown 2023-10-30T12:17:08.234 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 150704c8-4d6c-4df9-a728-25683a64db84 6 2023-10-30T12:17:08.250 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:08.249+0000 7fc4f022e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T12:17:08.321 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:08.320+0000 7f4e91df1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T12:17:08.334 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.332+0000 7f0d69461640 10 monclient: get_monmap_and_config 2023-10-30T12:17:08.334 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.332+0000 7f0d69461640 10 monclient: build_initial_monmap 2023-10-30T12:17:08.334 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.332+0000 7f0d69461640 10 monclient: monmap: 2023-10-30T12:17:08.335 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:08.335 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:08.335 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:17:08.333740+0000 2023-10-30T12:17:08.335 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:17:08.333740+0000 2023-10-30T12:17:08.335 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:08.336 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:08.336 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:08.336 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:08.336 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:08.336 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:08.337 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:08.337 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.333+0000 7f0d69461640 1 Processor -- start 2023-10-30T12:17:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.333+0000 7f0d69461640 1 -- start start 2023-10-30T12:17:08.340 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.333+0000 7f0d69461640 10 monclient: init 2023-10-30T12:17:08.340 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:08.340 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-10-30T12:17:08.340 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d640b7c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:08.340 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0d64141940 0x7f0d64141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.341 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 10 monclient(hunting): picked mon.noname-c con 0x7f0d64141940 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:08.341 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0d6413e730 0x7f0d6413eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.341 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 10 monclient(hunting): picked mon.noname-a con 0x7f0d6413e730 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:08.341 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d640a9350 con 0x7f0d640b7c10 2023-10-30T12:17:08.341 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.342 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d640670a0 con 0x7f0d6413e730 2023-10-30T12:17:08.342 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.342 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d640b7ff0 con 0x7f0d64141940 2023-10-30T12:17:08.342 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:08.342 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d69461640 10 monclient(hunting): authenticate will time out at 1524.565114s 2023-10-30T12:17:08.343 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d627fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0d64141940 0x7f0d64141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.343 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.334+0000 7f0d637fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0d6413e730 0x7f0d6413eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.343 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d62ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0d640b7c10 0x7f0d640a8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50614/0 (socket says 172.21.15.150:50614) 2023-10-30T12:17:08.343 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d62ffd640 1 -- 172.21.15.150:0/3803806219 learned_addr learned my addr 172.21.15.150:0/3803806219 (peer_addr_for_me v1:172.21.15.150:0/0) 2023-10-30T12:17:08.343 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d637fe640 10 monclient(hunting): get_auth_request con 0x7f0d6413e730 auth_method 0 2023-10-30T12:17:08.344 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.344 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.344 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.344 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d627fc640 10 monclient(hunting): get_auth_request con 0x7f0d64141940 auth_method 0 2023-10-30T12:17:08.344 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.344 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d627fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.345 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d627fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.345 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.345 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.345 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.335+0000 7f0d637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.345 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.336+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.346 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.336+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.346 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.336+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.346 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.337+0000 7f0d637fe640 10 monclient(hunting): handle_auth_done global_id 4146 payload 274 2023-10-30T12:17:08.346 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.337+0000 7f0d637fe640 1 -- 172.21.15.150:0/3803806219 >> v2:172.21.15.150:3300/0 conn(0x7f0d64141940 msgr2=0x7f0d64141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:08.346 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.337+0000 7f0d637fe640 1 --2- 172.21.15.150:0/3803806219 >> v2:172.21.15.150:3300/0 conn(0x7f0d64141940 0x7f0d64141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.347 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.337+0000 7f0d637fe640 1 -- 172.21.15.150:0/3803806219 >> v1:172.21.15.150:6789/0 conn(0x7f0d640b7c10 legacy=0x7f0d640a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:08.347 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.337+0000 7f0d637fe640 10 monclient: _finish_hunting 0 2023-10-30T12:17:08.347 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.337+0000 7f0d637fe640 1 monclient: found mon.noname-a 2023-10-30T12:17:08.347 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.337+0000 7f0d637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:08.347 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.337+0000 7f0d637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:17:08.348 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.337+0000 7f0d637fe640 1 -- 172.21.15.150:0/3803806219 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d640aa6a0 con 0x7f0d6413e730 2023-10-30T12:17:08.352 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.339+0000 7f0d637fe640 1 --2- 172.21.15.150:0/3803806219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d6413e730 0x7f0d6413eb80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0d640a9c40 tx=0x7f0d5c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b914d3bd293bb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:08.352 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 1 -- 172.21.15.150:0/3803806219 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c02cbf0 con 0x7f0d6413e730 2023-10-30T12:17:08.352 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:08.353 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:08.353 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 10 monclient: dump: 2023-10-30T12:17:08.353 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:08.353 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:08.353 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.354 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.354 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:08.354 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:08.354 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:08.354 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:08.355 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 10 monclient: _finish_auth 0 2023-10-30T12:17:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:17:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:08.356 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 1 -- 172.21.15.150:0/3803806219 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c02cd60 con 0x7f0d6413e730 2023-10-30T12:17:08.356 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:08.356 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 1 -- 172.21.15.150:0/3803806219 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c034660 con 0x7f0d6413e730 2023-10-30T12:17:08.356 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:08.356 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:08.357 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d61ffb640 10 monclient: dump: 2023-10-30T12:17:08.357 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:08.357 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:08.357 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.357 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.358 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:08.358 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:08.358 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:08.358 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:08.358 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:08.359 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.359 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d69461640 5 monclient: authenticate success, global_id 4146 2023-10-30T12:17:08.359 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d69461640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:08.359 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d69461640 4 set_mon_vals no callback set 2023-10-30T12:17:08.359 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.343+0000 7f0d69461640 10 monclient: shutdown 2023-10-30T12:17:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 1 -- 172.21.15.150:0/3803806219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d6413e730 msgr2=0x7f0d6413eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 1 --2- 172.21.15.150:0/3803806219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d6413e730 0x7f0d6413eb80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0d640a9c40 tx=0x7f0d5c02be40 comp rx=0 tx=0).stop 2023-10-30T12:17:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 1 -- 172.21.15.150:0/3803806219 shutdown_connections 2023-10-30T12:17:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 1 --2- 172.21.15.150:0/3803806219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d6413e730 0x7f0d6413eb80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.361 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 1 --2- 172.21.15.150:0/3803806219 >> v2:172.21.15.150:3300/0 conn(0x7f0d64141940 0x7f0d64141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.361 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 1 -- 172.21.15.150:0/3803806219 >> 172.21.15.150:0/3803806219 conn(0x7f0d6406c660 msgr2=0x7f0d640b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:08.361 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 1 -- 172.21.15.150:0/3803806219 shutdown_connections 2023-10-30T12:17:08.361 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 1 -- 172.21.15.150:0/3803806219 wait complete. 2023-10-30T12:17:08.361 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 10 monclient: build_initial_monmap 2023-10-30T12:17:08.362 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.344+0000 7f0d69461640 10 monclient: monmap: 2023-10-30T12:17:08.362 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:08.362 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:08.362 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2023-10-30T12:17:08.362 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2023-10-30T12:17:08.363 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:08.363 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2023-10-30T12:17:08.363 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:08.363 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:08.363 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:08.363 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 librados: starting msgr at 2023-10-30T12:17:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 librados: starting objecter 2023-10-30T12:17:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 Processor -- start 2023-10-30T12:17:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 -- start start 2023-10-30T12:17:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 librados: setting wanted keys 2023-10-30T12:17:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 librados: calling monclient init 2023-10-30T12:17:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient: init 2023-10-30T12:17:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:17:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d640b7c10 0x7f0d64139290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient(hunting): picked mon.noname-a con 0x7f0d640b7c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:08.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d6413e730 0x7f0d641da400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient(hunting): picked mon.noname-b con 0x7f0d6413e730 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:08.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0d64141940 0x7f0d640ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient(hunting): picked mon.noname-c con 0x7f0d64141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:08.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.367 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6413ee60 con 0x7f0d640b7c10 2023-10-30T12:17:08.367 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.367 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d640b0da0 con 0x7f0d6413e730 2023-10-30T12:17:08.367 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.368 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d640ad3a0 con 0x7f0d64141940 2023-10-30T12:17:08.368 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:08.368 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.345+0000 7f0d69461640 10 monclient(hunting): authenticate will time out at 1524.576877s 2023-10-30T12:17:08.368 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d62ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d640b7c10 0x7f0d64139290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.368 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d627fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d6413e730 0x7f0d641da400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.369 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d637fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0d64141940 0x7f0d640ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.369 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d62ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d640b7c10 0x7f0d64139290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.150:43768/0 (socket says 172.21.15.150:43768) 2023-10-30T12:17:08.369 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d62ffd640 1 -- 172.21.15.150:0/780197080 learned_addr learned my addr 172.21.15.150:0/780197080 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:08.369 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d62ffd640 10 monclient(hunting): get_auth_request con 0x7f0d640b7c10 auth_method 0 2023-10-30T12:17:08.369 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.370 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d62ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.370 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d62ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.370 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d627fc640 10 monclient(hunting): get_auth_request con 0x7f0d6413e730 auth_method 0 2023-10-30T12:17:08.370 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.370 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d627fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.370 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d627fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.371 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d637fe640 10 monclient(hunting): get_auth_request con 0x7f0d64141940 auth_method 0 2023-10-30T12:17:08.371 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.371 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.371 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.346+0000 7f0d637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.371 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.347+0000 7f0d62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.347+0000 7f0d62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.347+0000 7f0d62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.347+0000 7f0d637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.347+0000 7f0d637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.347+0000 7f0d637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.347+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.347+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.347+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.349+0000 7f0d62ffd640 10 monclient(hunting): handle_auth_done global_id 4149 payload 824 2023-10-30T12:17:08.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.349+0000 7f0d62ffd640 1 -- 172.21.15.150:0/780197080 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0d64141940 msgr2=0x7f0d640ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:08.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.350+0000 7f0d62ffd640 1 --2- 172.21.15.150:0/780197080 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0d64141940 0x7f0d640ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.350+0000 7f0d62ffd640 1 -- 172.21.15.150:0/780197080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d6413e730 msgr2=0x7f0d641da400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:08.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.350+0000 7f0d62ffd640 1 --2- 172.21.15.150:0/780197080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d6413e730 0x7f0d641da400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.350+0000 7f0d62ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:08.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.350+0000 7f0d62ffd640 1 monclient: found mon.noname-a 2023-10-30T12:17:08.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.350+0000 7f0d62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:08.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.350+0000 7f0d62ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:17:08.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.350+0000 7f0d62ffd640 1 -- 172.21.15.150:0/780197080 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d640aece0 con 0x7f0d640b7c10 2023-10-30T12:17:08.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.350+0000 7f0d627fc640 1 --2- 172.21.15.150:0/780197080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d6413e730 0x7f0d641da400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:17:08.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.352+0000 7f0d62ffd640 1 --2- 172.21.15.150:0/780197080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d640b7c10 0x7f0d64139290 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800c4a0 tx=0x7f0d5800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:08.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.352+0000 7f0d4b7fe640 10 client.?.objecter ms_handle_connect 0x7f0d640b7c10 2023-10-30T12:17:08.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.352+0000 7f0d4b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:08.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 1 -- 172.21.15.150:0/780197080 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58005ce0 con 0x7f0d640b7c10 2023-10-30T12:17:08.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:08.378 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:08.378 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 10 monclient: dump: 2023-10-30T12:17:08.378 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:08.378 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:08.378 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.379 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.379 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:08.379 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:08.379 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:08.379 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:08.380 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:08.380 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:08.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:08.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:08.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 1 -- 172.21.15.150:0/780197080 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5800e040 con 0x7f0d640b7c10 2023-10-30T12:17:08.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:08.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 1 -- 172.21.15.150:0/780197080 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58005e90 con 0x7f0d640b7c10 2023-10-30T12:17:08.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:08.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:08.382 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d4b7fe640 10 monclient: dump: 2023-10-30T12:17:08.382 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:08.382 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:08.382 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.382 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.382 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:08.383 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:08.383 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:08.383 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:08.383 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:08.383 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.384 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d69461640 5 monclient: authenticate success, global_id 4149 2023-10-30T12:17:08.384 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d69461640 10 monclient: _renew_subs 2023-10-30T12:17:08.384 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d69461640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:08.384 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d617fa640 4 set_mon_vals no callback set 2023-10-30T12:17:08.384 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d69461640 1 -- 172.21.15.150:0/780197080 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d641dab10 con 0x7f0d640b7c10 2023-10-30T12:17:08.385 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d69461640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:08.385 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d69461640 10 monclient: _renew_subs 2023-10-30T12:17:08.385 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d69461640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:08.385 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d69461640 1 -- 172.21.15.150:0/780197080 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d641dafb0 con 0x7f0d640b7c10 2023-10-30T12:17:08.385 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.356+0000 7f0d69461640 1 librados: init done 2023-10-30T12:17:08.385 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.357+0000 7f0d69461640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:08.386 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.357+0000 7f0d69461640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:08.386 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.357+0000 7f0d69461640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:08.386 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.357+0000 7f0d69461640 1 -- 172.21.15.150:0/780197080 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d640b2fd0 con 0x7f0d640b7c10 2023-10-30T12:17:08.386 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.359+0000 7f0d4b7fe640 1 -- 172.21.15.150:0/780197080 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0d58007570 con 0x7f0d640b7c10 2023-10-30T12:17:08.386 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.359+0000 7f0d4b7fe640 1 -- 172.21.15.150:0/780197080 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f0d5801b9a0 con 0x7f0d640b7c10 2023-10-30T12:17:08.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.359+0000 7f0d4b7fe640 10 client.4149.objecter ms_dispatch 0x7f0d64000b90 osd_map(8..8 src has 1..8) v4 2023-10-30T12:17:08.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.359+0000 7f0d4b7fe640 3 client.4149.objecter handle_osd_map got epochs [8,8] > 0 2023-10-30T12:17:08.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.359+0000 7f0d4b7fe640 3 client.4149.objecter handle_osd_map decoding full epoch 8 2023-10-30T12:17:08.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.359+0000 7f0d4b7fe640 20 client.4149.objecter dump_active .. 0 homeless 2023-10-30T12:17:08.388 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:08.380+0000 7fc4f022e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T12:17:08.426 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:08.425+0000 7fc4f022e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T12:17:08.457 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.456+0000 7f0d4b7fe640 1 -- 172.21.15.150:0/780197080 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f0d5801bbe0 con 0x7f0d640b7c10 2023-10-30T12:17:08.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.456+0000 7f0d4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:08.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.456+0000 7f0d4b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:08.524 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:08.523+0000 7fc4f022e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:17:08.528 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.527+0000 7f0d69461640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "150704c8-4d6c-4df9-a728-25683a64db84", "id": 6}] 2023-10-30T12:17:08.528 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.527+0000 7f0d69461640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "150704c8-4d6c-4df9-a728-25683a64db84", "id": 6}] 2023-10-30T12:17:08.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.527+0000 7f0d69461640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:08.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.527+0000 7f0d69461640 1 -- 172.21.15.150:0/780197080 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd new", "uuid": "150704c8-4d6c-4df9-a728-25683a64db84", "id": 6} v 0) v1 -- 0x7f0d640aa350 con 0x7f0d640b7c10 2023-10-30T12:17:08.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.558+0000 7f0d4b7fe640 1 -- 172.21.15.150:0/780197080 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "150704c8-4d6c-4df9-a728-25683a64db84", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0d5801b3b0 con 0x7f0d640b7c10 2023-10-30T12:17:08.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.558+0000 7f0d4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "150704c8-4d6c-4df9-a728-25683a64db84", "id": 6}] 2023-10-30T12:17:08.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.558+0000 7f0d4b7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:08.561 INFO:teuthology.orchestra.run.smithi150.stdout:6 2023-10-30T12:17:08.562 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 10 librados: watch_flush enter 2023-10-30T12:17:08.562 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 10 librados: watch_flush exit 2023-10-30T12:17:08.562 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 20 client.4149.objecter shutdown clearing up homeless session... 2023-10-30T12:17:08.563 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 10 client.4149.objecter successfully canceled tick 2023-10-30T12:17:08.563 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 10 monclient: shutdown 2023-10-30T12:17:08.563 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:08.563 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 1 -- 172.21.15.150:0/780197080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d640b7c10 msgr2=0x7f0d64139290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:08.563 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 1 --2- 172.21.15.150:0/780197080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d640b7c10 0x7f0d64139290 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800c4a0 tx=0x7f0d5800c960 comp rx=0 tx=0).stop 2023-10-30T12:17:08.564 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 1 -- 172.21.15.150:0/780197080 shutdown_connections 2023-10-30T12:17:08.564 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 1 --2- 172.21.15.150:0/780197080 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0d64141940 0x7f0d640ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.564 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 1 --2- 172.21.15.150:0/780197080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d6413e730 0x7f0d641da400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.564 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 1 --2- 172.21.15.150:0/780197080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d640b7c10 0x7f0d64139290 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.564 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 1 -- 172.21.15.150:0/780197080 >> 172.21.15.150:0/780197080 conn(0x7f0d6406c660 msgr2=0x7f0d6413aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:08.565 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 1 -- 172.21.15.150:0/780197080 shutdown_connections 2023-10-30T12:17:08.565 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 1 -- 172.21.15.150:0/780197080 wait complete. 2023-10-30T12:17:08.565 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.560+0000 7f0d69461640 1 librados: shutdown 2023-10-30T12:17:08.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 5daaf214-6591-4913-a103-5eb53ad2937b 7 2023-10-30T12:17:08.609 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:08.608+0000 7f4e91df1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T12:17:08.622 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:08.621+0000 7fc4f022e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T12:17:08.661 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:08.659+0000 7f4e91df1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T12:17:08.711 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.710+0000 7fd738ebb640 10 monclient: get_monmap_and_config 2023-10-30T12:17:08.712 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.710+0000 7fd738ebb640 10 monclient: build_initial_monmap 2023-10-30T12:17:08.712 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.710+0000 7fd738ebb640 10 monclient: monmap: 2023-10-30T12:17:08.712 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:08.712 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:08.713 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:17:08.711173+0000 2023-10-30T12:17:08.713 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:17:08.711173+0000 2023-10-30T12:17:08.713 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:08.713 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:08.713 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:08.714 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.710+0000 7fd738ebb640 1 Processor -- start 2023-10-30T12:17:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.710+0000 7fd738ebb640 1 -- start start 2023-10-30T12:17:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.710+0000 7fd738ebb640 10 monclient: init 2023-10-30T12:17:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T12:17:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd7340b7c10 0x7fd7340a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 10 monclient(hunting): picked mon.noname-a con 0x7fd7340b7c10 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd734141940 0x7fd734141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 10 monclient(hunting): picked mon.noname-b con 0x7fd734141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd7340b1c10 0x7fd7340b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 10 monclient(hunting): picked mon.noname-c con 0x7fd7340b1c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7340670a0 con 0x7fd7340b7c10 2023-10-30T12:17:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7340b7ff0 con 0x7fd7340b1c10 2023-10-30T12:17:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7340a9350 con 0x7fd734141940 2023-10-30T12:17:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.711+0000 7fd738ebb640 10 monclient(hunting): authenticate will time out at 1524.942682s 2023-10-30T12:17:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd732d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd7340b1c10 0x7fd7340b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd73259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd7340b7c10 0x7fd7340a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd731d9b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd734141940 0x7fd734141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.722 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd732d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd7340b1c10 0x7fd7340b2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38704/0 (socket says 172.21.15.150:38704) 2023-10-30T12:17:08.722 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd732d9d640 1 -- 172.21.15.150:0/3759039133 learned_addr learned my addr 172.21.15.150:0/3759039133 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:08.722 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd732d9d640 10 monclient(hunting): get_auth_request con 0x7fd7340b1c10 auth_method 0 2023-10-30T12:17:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd732d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd732d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd732d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd73259c640 10 monclient(hunting): get_auth_request con 0x7fd7340b7c10 auth_method 0 2023-10-30T12:17:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd73259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd73259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.712+0000 7fd73259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd731d9b640 10 monclient(hunting): get_auth_request con 0x7fd734141940 auth_method 0 2023-10-30T12:17:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd731d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd731d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd731d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd732d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd732d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd732d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd73259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd73259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd73259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd731d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd731d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.713+0000 7fd731d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.714+0000 7fd732d9d640 10 monclient(hunting): handle_auth_done global_id 4153 payload 274 2023-10-30T12:17:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.714+0000 7fd732d9d640 1 -- 172.21.15.150:0/3759039133 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd734141940 msgr2=0x7fd734141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:08.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.714+0000 7fd732d9d640 1 --2- 172.21.15.150:0/3759039133 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd734141940 0x7fd734141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.714+0000 7fd732d9d640 1 -- 172.21.15.150:0/3759039133 >> v2:172.21.15.148:3300/0 conn(0x7fd7340b7c10 msgr2=0x7fd7340a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:08.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.714+0000 7fd732d9d640 1 --2- 172.21.15.150:0/3759039133 >> v2:172.21.15.148:3300/0 conn(0x7fd7340b7c10 0x7fd7340a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.714+0000 7fd732d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:17:08.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.714+0000 7fd732d9d640 1 monclient: found mon.noname-c 2023-10-30T12:17:08.728 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.714+0000 7fd732d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:08.728 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.714+0000 7fd732d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:17:08.728 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.714+0000 7fd732d9d640 1 -- 172.21.15.150:0/3759039133 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7340aa6a0 con 0x7fd7340b1c10 2023-10-30T12:17:08.728 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.716+0000 7fd732d9d640 1 --2- 172.21.15.150:0/3759039133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7340b1c10 0x7fd7340b2060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd7340b3c70 tx=0x7fd72c02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4f9baa99ab87eea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:08.732 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 1 -- 172.21.15.150:0/3759039133 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c02c9e0 con 0x7fd7340b1c10 2023-10-30T12:17:08.732 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:08.732 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:08.732 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 10 monclient: dump: 2023-10-30T12:17:08.733 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:08.733 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:08.733 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.733 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.733 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:08.734 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:08.734 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:08.734 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:08.734 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:08.734 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 10 monclient: _finish_auth 0 2023-10-30T12:17:08.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 10 monclient: _check_auth_tickets 2023-10-30T12:17:08.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:08.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 1 -- 172.21.15.150:0/3759039133 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72c02cb50 con 0x7fd7340b1c10 2023-10-30T12:17:08.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:08.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 1 -- 172.21.15.150:0/3759039133 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c0343f0 con 0x7fd7340b1c10 2023-10-30T12:17:08.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd738ebb640 5 monclient: authenticate success, global_id 4153 2023-10-30T12:17:08.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd738ebb640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:08.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd738ebb640 4 set_mon_vals no callback set 2023-10-30T12:17:08.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:08.736 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:08.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd73159a640 10 monclient: dump: 2023-10-30T12:17:08.737 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:08.737 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:08.737 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.737 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.738 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:08.738 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:08.738 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:08.738 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:08.738 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:08.738 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd738ebb640 10 monclient: shutdown 2023-10-30T12:17:08.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd738ebb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:08.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd738ebb640 1 -- 172.21.15.150:0/3759039133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7340b1c10 msgr2=0x7fd7340b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:08.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.720+0000 7fd738ebb640 1 --2- 172.21.15.150:0/3759039133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7340b1c10 0x7fd7340b2060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd7340b3c70 tx=0x7fd72c02be40 comp rx=0 tx=0).stop 2023-10-30T12:17:08.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.721+0000 7fd738ebb640 1 -- 172.21.15.150:0/3759039133 shutdown_connections 2023-10-30T12:17:08.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.721+0000 7fd738ebb640 1 --2- 172.21.15.150:0/3759039133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7340b1c10 0x7fd7340b2060 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.721+0000 7fd738ebb640 1 --2- 172.21.15.150:0/3759039133 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd734141940 0x7fd734141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.721+0000 7fd738ebb640 1 --2- 172.21.15.150:0/3759039133 >> v2:172.21.15.148:3300/0 conn(0x7fd7340b7c10 0x7fd7340a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.721+0000 7fd738ebb640 1 -- 172.21.15.150:0/3759039133 >> 172.21.15.150:0/3759039133 conn(0x7fd73406c660 msgr2=0x7fd7340b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:08.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.721+0000 7fd738ebb640 1 -- 172.21.15.150:0/3759039133 shutdown_connections 2023-10-30T12:17:08.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.721+0000 7fd738ebb640 1 -- 172.21.15.150:0/3759039133 wait complete. 2023-10-30T12:17:08.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.721+0000 7fd738ebb640 10 monclient: build_initial_monmap 2023-10-30T12:17:08.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.721+0000 7fd738ebb640 10 monclient: monmap: 2023-10-30T12:17:08.741 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2023-10-30T12:17:08.741 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:08.741 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2023-10-30T12:17:08.742 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2023-10-30T12:17:08.742 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:08.742 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2023-10-30T12:17:08.742 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:08.742 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:08.743 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:08.743 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.743 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 librados: starting msgr at 2023-10-30T12:17:08.743 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 librados: starting objecter 2023-10-30T12:17:08.743 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 Processor -- start 2023-10-30T12:17:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 -- start start 2023-10-30T12:17:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 librados: setting wanted keys 2023-10-30T12:17:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 librados: calling monclient init 2023-10-30T12:17:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient: init 2023-10-30T12:17:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:17:08.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7340b1c10 0x7fd73413a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient(hunting): picked mon.noname-a con 0x7fd7340b1c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:08.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd7340b7c10 0x7fd7340b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient(hunting): picked mon.noname-c con 0x7fd7340b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:08.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd734141940 0x7fd73411f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:08.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient(hunting): picked mon.noname-b con 0x7fd734141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:08.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd734141f00 con 0x7fd7340b1c10 2023-10-30T12:17:08.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73411fa60 con 0x7fd734141940 2023-10-30T12:17:08.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:08.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73413b030 con 0x7fd7340b7c10 2023-10-30T12:17:08.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:08.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.722+0000 7fd738ebb640 10 monclient(hunting): authenticate will time out at 1524.953956s 2023-10-30T12:17:08.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd732d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd734141940 0x7fd73411f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd73259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7340b1c10 0x7fd73413a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd732d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd734141940 0x7fd73411f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38720/0 (socket says 172.21.15.150:38720) 2023-10-30T12:17:08.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd732d9d640 1 -- 172.21.15.150:0/1981149910 learned_addr learned my addr 172.21.15.150:0/1981149910 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T12:17:08.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd732d9d640 10 monclient(hunting): get_auth_request con 0x7fd734141940 auth_method 0 2023-10-30T12:17:08.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd732d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd732d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd732d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd731d9b640 1 --2- 172.21.15.150:0/1981149910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd7340b7c10 0x7fd7340b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:08.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd73259c640 10 monclient(hunting): get_auth_request con 0x7fd7340b1c10 auth_method 0 2023-10-30T12:17:08.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd73259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd73259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.723+0000 7fd73259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd731d9b640 10 monclient(hunting): get_auth_request con 0x7fd7340b7c10 auth_method 0 2023-10-30T12:17:08.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd731d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:08.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd731d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:08.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd731d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:08.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd732d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd732d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd732d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd73259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd73259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd73259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd731d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:08.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd731d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:08.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.724+0000 7fd731d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:08.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.727+0000 7fd732d9d640 10 monclient(hunting): handle_auth_done global_id 4156 payload 824 2023-10-30T12:17:08.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.727+0000 7fd732d9d640 1 -- 172.21.15.150:0/1981149910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd7340b7c10 msgr2=0x7fd7340b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:08.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.727+0000 7fd732d9d640 1 --2- 172.21.15.150:0/1981149910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd7340b7c10 0x7fd7340b0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.727+0000 7fd732d9d640 1 -- 172.21.15.150:0/1981149910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7340b1c10 msgr2=0x7fd73413a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:08.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.727+0000 7fd732d9d640 1 --2- 172.21.15.150:0/1981149910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7340b1c10 0x7fd73413a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:08.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.727+0000 7fd732d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:17:08.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.727+0000 7fd732d9d640 1 monclient: found mon.noname-b 2023-10-30T12:17:08.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.727+0000 7fd732d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:08.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.727+0000 7fd732d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:08.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.727+0000 7fd732d9d640 1 -- 172.21.15.150:0/1981149910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7340aff80 con 0x7fd734141940 2023-10-30T12:17:08.756 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.729+0000 7fd732d9d640 1 --2- 172.21.15.150:0/1981149910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd734141940 0x7fd73411f140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd72c036040 tx=0x7fd72c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:08.756 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.729+0000 7fd722ffd640 10 client.?.objecter ms_handle_connect 0x7fd734141940 2023-10-30T12:17:08.756 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.729+0000 7fd722ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 1 -- 172.21.15.150:0/1981149910 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c009510 con 0x7fd734141940 2023-10-30T12:17:08.759 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:08.759 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:08.759 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 10 monclient: dump: 2023-10-30T12:17:08.759 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:08.759 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:08.761 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 10 monclient: _finish_auth 0 2023-10-30T12:17:08.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:08.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:08.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 1 -- 172.21.15.150:0/1981149910 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72c03e450 con 0x7fd734141940 2023-10-30T12:17:08.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd738ebb640 5 monclient: authenticate success, global_id 4156 2023-10-30T12:17:08.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd738ebb640 10 monclient: _renew_subs 2023-10-30T12:17:08.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd738ebb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:08.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd738ebb640 1 -- 172.21.15.150:0/1981149910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73411f680 con 0x7fd734141940 2023-10-30T12:17:08.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:08.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 1 -- 172.21.15.150:0/1981149910 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c03e810 con 0x7fd734141940 2023-10-30T12:17:08.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd722ffd640 10 monclient: dump: 2023-10-30T12:17:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2023-10-30T12:17:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 3 2023-10-30T12:17:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:08.766 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:08.766 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:17:08.766 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd730d99640 4 set_mon_vals no callback set 2023-10-30T12:17:08.766 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd738ebb640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:08.766 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd738ebb640 10 monclient: _renew_subs 2023-10-30T12:17:08.766 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd738ebb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:08.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd738ebb640 1 -- 172.21.15.150:0/1981149910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7341da3b0 con 0x7fd734141940 2023-10-30T12:17:08.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.733+0000 7fd738ebb640 1 librados: init done 2023-10-30T12:17:08.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.734+0000 7fd738ebb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:08.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.734+0000 7fd738ebb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:08.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.734+0000 7fd738ebb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:08.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.734+0000 7fd738ebb640 1 -- 172.21.15.150:0/1981149910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73411fbb0 con 0x7fd734141940 2023-10-30T12:17:08.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.737+0000 7fd722ffd640 1 -- 172.21.15.150:0/1981149910 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd72c03ec80 con 0x7fd734141940 2023-10-30T12:17:08.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.737+0000 7fd722ffd640 1 -- 172.21.15.150:0/1981149910 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fd72c03fa60 con 0x7fd734141940 2023-10-30T12:17:08.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.737+0000 7fd722ffd640 10 client.4156.objecter ms_dispatch 0x7fd734000b90 osd_map(9..9 src has 1..9) v4 2023-10-30T12:17:08.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.737+0000 7fd722ffd640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2023-10-30T12:17:08.769 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.737+0000 7fd722ffd640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2023-10-30T12:17:08.769 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.737+0000 7fd722ffd640 20 client.4156.objecter dump_active .. 0 homeless 2023-10-30T12:17:08.769 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:08.763+0000 7f4e91df1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T12:17:08.771 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:08.770+0000 7fc4f022e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:17:08.814 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:08.813+0000 7fc4f022e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T12:17:08.836 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.835+0000 7fd722ffd640 1 -- 172.21.15.150:0/1981149910 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd72c02c9f0 con 0x7fd734141940 2023-10-30T12:17:08.836 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.835+0000 7fd722ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:08.837 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.835+0000 7fd722ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:08.908 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.907+0000 7fd738ebb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5daaf214-6591-4913-a103-5eb53ad2937b", "id": 7}] 2023-10-30T12:17:08.908 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.907+0000 7fd738ebb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5daaf214-6591-4913-a103-5eb53ad2937b", "id": 7}] 2023-10-30T12:17:08.909 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.907+0000 7fd738ebb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:08.909 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:08.907+0000 7fd738ebb640 1 -- 172.21.15.150:0/1981149910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5daaf214-6591-4913-a103-5eb53ad2937b", "id": 7} v 0) v1 -- 0x7fd73411f820 con 0x7fd734141940 2023-10-30T12:17:08.960 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:08.958+0000 7fc4f022e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T12:17:09.053 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.052+0000 7fd722ffd640 1 -- 172.21.15.150:0/1981149910 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7fd72c03fca0 con 0x7fd734141940 2023-10-30T12:17:09.324 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.323+0000 7fd722ffd640 1 -- 172.21.15.150:0/1981149910 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5daaf214-6591-4913-a103-5eb53ad2937b", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd72c032300 con 0x7fd734141940 2023-10-30T12:17:09.325 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.323+0000 7fd722ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5daaf214-6591-4913-a103-5eb53ad2937b", "id": 7}] 2023-10-30T12:17:09.325 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.323+0000 7fd722ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:09.325 INFO:teuthology.orchestra.run.smithi150.stdout:7 2023-10-30T12:17:09.327 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 10 librados: watch_flush enter 2023-10-30T12:17:09.327 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 10 librados: watch_flush exit 2023-10-30T12:17:09.327 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 20 client.4156.objecter shutdown clearing up homeless session... 2023-10-30T12:17:09.328 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 10 client.4156.objecter successfully canceled tick 2023-10-30T12:17:09.328 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 10 monclient: shutdown 2023-10-30T12:17:09.328 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:09.328 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 1 -- 172.21.15.150:0/1981149910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd734141940 msgr2=0x7fd73411f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:09.328 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 1 --2- 172.21.15.150:0/1981149910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd734141940 0x7fd73411f140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd72c036040 tx=0x7fd72c004a40 comp rx=0 tx=0).stop 2023-10-30T12:17:09.329 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 1 -- 172.21.15.150:0/1981149910 shutdown_connections 2023-10-30T12:17:09.329 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 1 --2- 172.21.15.150:0/1981149910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd734141940 0x7fd73411f140 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:09.329 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 1 --2- 172.21.15.150:0/1981149910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd7340b7c10 0x7fd7340b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:09.329 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 1 --2- 172.21.15.150:0/1981149910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7340b1c10 0x7fd73413a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:09.330 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 1 -- 172.21.15.150:0/1981149910 >> 172.21.15.150:0/1981149910 conn(0x7fd73406c660 msgr2=0x7fd73413f840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:09.330 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 1 -- 172.21.15.150:0/1981149910 shutdown_connections 2023-10-30T12:17:09.330 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 1 -- 172.21.15.150:0/1981149910 wait complete. 2023-10-30T12:17:09.330 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T12:17:09.324+0000 7fd738ebb640 1 librados: shutdown 2023-10-30T12:17:09.336 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:17:09.337 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T12:17:09.338 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-10-30T12:17:09.340 INFO:tasks.ceph.osd.0:Started 2023-10-30T12:17:09.341 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:17:09.341 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T12:17:09.342 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-10-30T12:17:09.344 INFO:tasks.ceph.osd.1:Started 2023-10-30T12:17:09.344 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:17:09.345 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T12:17:09.345 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-10-30T12:17:09.348 INFO:tasks.ceph.osd.2:Started 2023-10-30T12:17:09.348 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:17:09.348 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T12:17:09.349 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-10-30T12:17:09.351 INFO:tasks.ceph.osd.3:Started 2023-10-30T12:17:09.352 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:17:09.352 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T12:17:09.353 DEBUG:teuthology.orchestra.run.smithi150:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-10-30T12:17:09.379 INFO:tasks.ceph.osd.4:Started 2023-10-30T12:17:09.379 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:17:09.379 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T12:17:09.380 DEBUG:teuthology.orchestra.run.smithi150:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-10-30T12:17:09.382 INFO:tasks.ceph.osd.5:Started 2023-10-30T12:17:09.382 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:17:09.383 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-30T12:17:09.383 DEBUG:teuthology.orchestra.run.smithi150:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-10-30T12:17:09.386 INFO:tasks.ceph.osd.6:Started 2023-10-30T12:17:09.387 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:17:09.387 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-30T12:17:09.387 DEBUG:teuthology.orchestra.run.smithi150:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-10-30T12:17:09.390 INFO:tasks.ceph.osd.7:Started 2023-10-30T12:17:09.390 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:17:09.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.514+0000 7f7290856640 10 monclient: get_monmap_and_config 2023-10-30T12:17:09.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.514+0000 7f7290856640 10 monclient: build_initial_monmap 2023-10-30T12:17:09.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.514+0000 7f7290856640 10 monclient: monmap: 2023-10-30T12:17:09.517 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:17:09.517 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:09.517 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:17:09.515227+0000 2023-10-30T12:17:09.518 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:17:09.515227+0000 2023-10-30T12:17:09.518 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:09.518 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:09.518 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:09.519 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:09.519 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:09.519 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:09.519 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:09.520 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:09.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.514+0000 7f7290856640 1 Processor -- start 2023-10-30T12:17:09.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.514+0000 7f7290856640 1 -- start start 2023-10-30T12:17:09.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.514+0000 7f7290856640 10 monclient: init 2023-10-30T12:17:09.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:09.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-10-30T12:17:09.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f728c0b7c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:09.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c141940 0x7f728c141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:09.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 10 monclient(hunting): picked mon.noname-b con 0x7f728c141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:09.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f728c13e730 0x7f728c13eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:09.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 10 monclient(hunting): picked mon.noname-c con 0x7f728c13e730 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:09.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728c0a9350 con 0x7f728c0b7c10 2023-10-30T12:17:09.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:09.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f728c0670a0 con 0x7f728c13e730 2023-10-30T12:17:09.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:09.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728c0b7ff0 con 0x7f728c141940 2023-10-30T12:17:09.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:09.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.515+0000 7f7290856640 10 monclient(hunting): authenticate will time out at 1521.720816s 2023-10-30T12:17:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.516+0000 7f7289d9b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c141940 0x7f728c141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.516+0000 7f7289d9b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c141940 0x7f728c141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:53072/0 (socket says 172.21.15.148:53072) 2023-10-30T12:17:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.516+0000 7f7289d9b640 1 -- 172.21.15.148:0/1219306806 learned_addr learned my addr 172.21.15.148:0/1219306806 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:09.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.516+0000 7f7289d9b640 10 monclient(hunting): get_auth_request con 0x7f728c141940 auth_method 0 2023-10-30T12:17:09.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.516+0000 7f7289d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:09.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.516+0000 7f7289d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:09.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.516+0000 7f7289d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:09.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.517+0000 7f728ad9d640 1 --2- 172.21.15.148:0/1219306806 >> v2:172.21.15.150:3300/0 conn(0x7f728c13e730 0x7f728c13eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:09.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.517+0000 7f7289d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:09.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.517+0000 7f7289d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:09.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.517+0000 7f7289d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:09.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.517+0000 7f728ad9d640 10 monclient(hunting): get_auth_request con 0x7f728c13e730 auth_method 0 2023-10-30T12:17:09.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.517+0000 7f728ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:09.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.517+0000 7f728ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:09.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.517+0000 7f728ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:09.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.518+0000 7f728ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:09.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.518+0000 7f728ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:09.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.518+0000 7f728ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:09.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.520+0000 7f7289d9b640 10 monclient(hunting): handle_auth_done global_id 4154 payload 274 2023-10-30T12:17:09.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.520+0000 7f7289d9b640 1 -- 172.21.15.148:0/1219306806 >> v2:172.21.15.150:3300/0 conn(0x7f728c13e730 msgr2=0x7f728c13eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:09.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.520+0000 7f7289d9b640 1 --2- 172.21.15.148:0/1219306806 >> v2:172.21.15.150:3300/0 conn(0x7f728c13e730 0x7f728c13eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:09.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.520+0000 7f7289d9b640 1 -- 172.21.15.148:0/1219306806 >> v1:172.21.15.150:6789/0 conn(0x7f728c0b7c10 legacy=0x7f728c0a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:09.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.520+0000 7f7289d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:17:09.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.520+0000 7f7289d9b640 1 monclient: found mon.noname-b 2023-10-30T12:17:09.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.520+0000 7f7289d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:09.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.520+0000 7f7289d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:17:09.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.520+0000 7f7289d9b640 1 -- 172.21.15.148:0/1219306806 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c0aa6a0 con 0x7f728c141940 2023-10-30T12:17:09.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.523+0000 7f7289d9b640 1 --2- 172.21.15.148:0/1219306806 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c141940 0x7f728c141d60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7280009870 tx=0x7f728002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=670ea2d847914c64 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:09.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.526+0000 7f728959a640 1 -- 172.21.15.148:0/1219306806 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280031640 con 0x7f728c141940 2023-10-30T12:17:09.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.526+0000 7f728959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:09.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.526+0000 7f728959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:09.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.526+0000 7f728959a640 10 monclient: dump: 2023-10-30T12:17:09.539 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:09.539 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:09.539 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:09.539 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:09.539 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:09.540 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:09.540 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:09.540 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:09.540 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:09.540 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:09.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.526+0000 7f728959a640 10 monclient: _finish_auth 0 2023-10-30T12:17:09.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.526+0000 7f728959a640 10 monclient: _check_auth_tickets 2023-10-30T12:17:09.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.526+0000 7f728959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:09.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.526+0000 7f7290856640 5 monclient: authenticate success, global_id 4154 2023-10-30T12:17:09.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.526+0000 7f7290856640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:09.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.529+0000 7f728959a640 1 -- 172.21.15.148:0/1219306806 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72800317b0 con 0x7f728c141940 2023-10-30T12:17:09.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.529+0000 7f728959a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:09.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.529+0000 7f728959a640 1 -- 172.21.15.148:0/1219306806 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280031b50 con 0x7f728c141940 2023-10-30T12:17:09.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.529+0000 7f7290856640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:09.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.529+0000 7f7290856640 4 set_mon_vals no callback set 2023-10-30T12:17:09.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.529+0000 7f728959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:09.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.529+0000 7f728959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:09.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.529+0000 7f728959a640 10 monclient: dump: 2023-10-30T12:17:09.544 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:09.544 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:09.544 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:09.544 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:09.545 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:09.545 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:09.545 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:09.545 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:09.546 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:09.546 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:09.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.530+0000 7f7290856640 10 monclient: shutdown 2023-10-30T12:17:09.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.530+0000 7f7290856640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:09.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.530+0000 7f7290856640 1 -- 172.21.15.148:0/1219306806 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c141940 msgr2=0x7f728c141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:09.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.530+0000 7f7290856640 1 --2- 172.21.15.148:0/1219306806 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c141940 0x7f728c141d60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7280009870 tx=0x7f728002b8f0 comp rx=0 tx=0).stop 2023-10-30T12:17:09.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.530+0000 7f7290856640 1 -- 172.21.15.148:0/1219306806 shutdown_connections 2023-10-30T12:17:09.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.530+0000 7f7290856640 1 --2- 172.21.15.148:0/1219306806 >> v2:172.21.15.150:3300/0 conn(0x7f728c13e730 0x7f728c13eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:09.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.530+0000 7f7290856640 1 --2- 172.21.15.148:0/1219306806 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c141940 0x7f728c141d60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:09.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.530+0000 7f7290856640 1 -- 172.21.15.148:0/1219306806 >> 172.21.15.148:0/1219306806 conn(0x7f728c06c660 msgr2=0x7f728c0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:09.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.530+0000 7f7290856640 1 -- 172.21.15.148:0/1219306806 shutdown_connections 2023-10-30T12:17:09.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.530+0000 7f7290856640 1 -- 172.21.15.148:0/1219306806 wait complete. 2023-10-30T12:17:09.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.531+0000 7f7290856640 10 monclient: build_initial_monmap 2023-10-30T12:17:09.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.531+0000 7f7290856640 10 monclient: monmap: 2023-10-30T12:17:09.549 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:17:09.549 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:09.549 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:09.550 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:17:09.550 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:09.550 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:09.550 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:09.551 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:09.551 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:09.551 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:09.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.531+0000 7f7290856640 1 librados: starting msgr at 2023-10-30T12:17:09.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.531+0000 7f7290856640 1 librados: starting objecter 2023-10-30T12:17:09.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.531+0000 7f7290856640 1 Processor -- start 2023-10-30T12:17:09.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 1 -- start start 2023-10-30T12:17:09.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 1 librados: setting wanted keys 2023-10-30T12:17:09.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 1 librados: calling monclient init 2023-10-30T12:17:09.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient: init 2023-10-30T12:17:09.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:09.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:17:09.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c0b7c10 0x7f728c1da4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:09.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient(hunting): picked mon.noname-c con 0x7f728c0b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:09.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c13e730 0x7f728c1c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:09.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient(hunting): picked mon.noname-b con 0x7f728c13e730 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:09.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f728c141940 0x7f728c13a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:09.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient(hunting): picked mon.noname-a con 0x7f728c141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:09.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:09.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c13f4e0 con 0x7f728c141940 2023-10-30T12:17:09.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:09.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c1c8180 con 0x7f728c13e730 2023-10-30T12:17:09.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728c1c9a10 con 0x7f728c0b7c10 2023-10-30T12:17:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.532+0000 7f7290856640 10 monclient(hunting): authenticate will time out at 1521.737790s 2023-10-30T12:17:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.533+0000 7f728a59c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c0b7c10 0x7f728c1da4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:09.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.533+0000 7f728a59c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c0b7c10 0x7f728c1da4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:53080/0 (socket says 172.21.15.148:53080) 2023-10-30T12:17:09.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.533+0000 7f728a59c640 1 -- 172.21.15.148:0/873314118 learned_addr learned my addr 172.21.15.148:0/873314118 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:09.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.533+0000 7f728a59c640 10 monclient(hunting): get_auth_request con 0x7f728c0b7c10 auth_method 0 2023-10-30T12:17:09.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.533+0000 7f728a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.533+0000 7f728a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.533+0000 7f728a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.533+0000 7f7289d9b640 1 --2- 172.21.15.148:0/873314118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c13e730 0x7f728c1c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.534+0000 7f7289d9b640 10 monclient(hunting): get_auth_request con 0x7f728c13e730 auth_method 0 2023-10-30T12:17:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.534+0000 7f7289d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.534+0000 7f7289d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.534+0000 7f7289d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.534+0000 7f728a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.534+0000 7f728a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.534+0000 7f728a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.535+0000 7f7289d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.535+0000 7f7289d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.535+0000 7f7289d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:09.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.539+0000 7f7289d9b640 10 monclient(hunting): handle_auth_done global_id 4174 payload 824 2023-10-30T12:17:09.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.539+0000 7f7289d9b640 1 -- 172.21.15.148:0/873314118 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c0b7c10 msgr2=0x7f728c1da4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:09.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.539+0000 7f7289d9b640 1 --2- 172.21.15.148:0/873314118 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c0b7c10 0x7f728c1da4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:09.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.539+0000 7f7289d9b640 1 -- 172.21.15.148:0/873314118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f728c141940 msgr2=0x7f728c13a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:09.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.539+0000 7f7289d9b640 1 --2- 172.21.15.148:0/873314118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f728c141940 0x7f728c13a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:09.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.539+0000 7f7289d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:17:09.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.539+0000 7f7289d9b640 1 monclient: found mon.noname-b 2023-10-30T12:17:09.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.539+0000 7f7289d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:09.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.539+0000 7f7289d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:09.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.539+0000 7f7289d9b640 1 -- 172.21.15.148:0/873314118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c0b0220 con 0x7f728c13e730 2023-10-30T12:17:09.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.546+0000 7f7289d9b640 1 --2- 172.21.15.148:0/873314118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c13e730 0x7f728c1c7ae0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7280033000 tx=0x7f72800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:09.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.546+0000 7f727affd640 10 client.?.objecter ms_handle_connect 0x7f728c13e730 2023-10-30T12:17:09.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.546+0000 7f727affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 1 -- 172.21.15.148:0/873314118 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728003abe0 con 0x7f728c13e730 2023-10-30T12:17:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 10 monclient: dump: 2023-10-30T12:17:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:09.624 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 10 monclient: _finish_auth 0 2023-10-30T12:17:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 1 -- 172.21.15.148:0/873314118 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728003e030 con 0x7f728c13e730 2023-10-30T12:17:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 1 -- 172.21.15.148:0/873314118 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280031e30 con 0x7f728c13e730 2023-10-30T12:17:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:09.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f727affd640 10 monclient: dump: 2023-10-30T12:17:09.626 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:09.626 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:09.626 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:09.626 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:09.627 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:09.627 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:09.627 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:09.627 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:09.627 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:09.628 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:09.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f7290856640 5 monclient: authenticate success, global_id 4174 2023-10-30T12:17:09.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f7290856640 10 monclient: _renew_subs 2023-10-30T12:17:09.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f7290856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:09.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f7290856640 1 -- 172.21.15.148:0/873314118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728c13b120 con 0x7f728c13e730 2023-10-30T12:17:09.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.619+0000 7f7288d99640 4 set_mon_vals no callback set 2023-10-30T12:17:09.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.621+0000 7f7290856640 10 client.4174.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:09.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.621+0000 7f7290856640 10 monclient: _renew_subs 2023-10-30T12:17:09.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.621+0000 7f7290856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:09.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.621+0000 7f7290856640 1 -- 172.21.15.148:0/873314118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728c13a5c0 con 0x7f728c13e730 2023-10-30T12:17:09.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.621+0000 7f7290856640 1 librados: init done 2023-10-30T12:17:09.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.622+0000 7f7290856640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:09.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.622+0000 7f7290856640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:09.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.622+0000 7f7290856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:09.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.622+0000 7f7290856640 1 -- 172.21.15.148:0/873314118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728c0aa350 con 0x7f728c13e730 2023-10-30T12:17:09.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.638+0000 7f727affd640 1 -- 172.21.15.148:0/873314118 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7f728002f050 con 0x7f728c13e730 2023-10-30T12:17:09.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.642+0000 7f727affd640 1 -- 172.21.15.148:0/873314118 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7280052080 con 0x7f728c13e730 2023-10-30T12:17:09.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.642+0000 7f727affd640 10 client.4174.objecter ms_dispatch 0x7f728c000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:17:09.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.642+0000 7f727affd640 3 client.4174.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:17:09.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.642+0000 7f727affd640 3 client.4174.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:17:09.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.642+0000 7f727affd640 20 client.4174.objecter dump_active .. 0 homeless 2023-10-30T12:17:09.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.839+0000 7f727affd640 1 -- 172.21.15.148:0/873314118 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f72800433a0 con 0x7f728c13e730 2023-10-30T12:17:09.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.839+0000 7f727affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:09.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.839+0000 7f727affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:09.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.957+0000 7f7290856640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:09.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.957+0000 7f7290856640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:09.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.957+0000 7f7290856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:09.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:09.957+0000 7f7290856640 1 -- 172.21.15.148:0/873314118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f728c0aa6a0 con 0x7f728c13e730 2023-10-30T12:17:09.991 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:09.990+0000 7fc4f022e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T12:17:10.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.129+0000 7f727affd640 1 -- 172.21.15.148:0/873314118 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f7280030e00 con 0x7f728c13e730 2023-10-30T12:17:10.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.129+0000 7f727affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:10.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.129+0000 7f727affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:10.132 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:17:10.133 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":10,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:17:09.070671+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:17:10.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 10 librados: watch_flush enter 2023-10-30T12:17:10.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 10 librados: watch_flush exit 2023-10-30T12:17:10.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 20 client.4174.objecter shutdown clearing up homeless session... 2023-10-30T12:17:10.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 10 client.4174.objecter successfully canceled tick 2023-10-30T12:17:10.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 10 monclient: shutdown 2023-10-30T12:17:10.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:10.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 1 -- 172.21.15.148:0/873314118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c13e730 msgr2=0x7f728c1c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:10.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 1 --2- 172.21.15.148:0/873314118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c13e730 0x7f728c1c7ae0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7280033000 tx=0x7f72800046c0 comp rx=0 tx=0).stop 2023-10-30T12:17:10.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 1 -- 172.21.15.148:0/873314118 shutdown_connections 2023-10-30T12:17:10.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 1 --2- 172.21.15.148:0/873314118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f728c141940 0x7f728c13a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:10.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 1 --2- 172.21.15.148:0/873314118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c13e730 0x7f728c1c7ae0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:10.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 1 --2- 172.21.15.148:0/873314118 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f728c0b7c10 0x7f728c1da4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:10.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 1 -- 172.21.15.148:0/873314118 >> 172.21.15.148:0/873314118 conn(0x7f728c06c660 msgr2=0x7f728c13c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:10.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 1 -- 172.21.15.148:0/873314118 shutdown_connections 2023-10-30T12:17:10.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 1 -- 172.21.15.148:0/873314118 wait complete. 2023-10-30T12:17:10.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:10.131+0000 7f7290856640 1 librados: shutdown 2023-10-30T12:17:10.148 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T12:17:10.148 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T12:17:10.474 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:10.473+0000 7fc4f022e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T12:17:10.562 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:10.560+0000 7fc4f022e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T12:17:10.749 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T12:17:10.747+0000 7fc4f022e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T12:17:10.955 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:17:11.107 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.106+0000 7eff9d3ee640 10 monclient: get_monmap_and_config 2023-10-30T12:17:11.108 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.106+0000 7eff9d3ee640 10 monclient: build_initial_monmap 2023-10-30T12:17:11.108 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.106+0000 7eff9d3ee640 10 monclient: monmap: 2023-10-30T12:17:11.108 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:11.108 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:11.109 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:17:11.107299+0000 2023-10-30T12:17:11.109 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:17:11.107299+0000 2023-10-30T12:17:11.109 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:11.109 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:11.109 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:11.110 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:11.110 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:11.110 INFO:teuthology.misc.health.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:11.110 INFO:teuthology.misc.health.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:11.110 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:11.112 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.107+0000 7eff9d3ee640 1 Processor -- start 2023-10-30T12:17:11.112 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.107+0000 7eff9d3ee640 1 -- start start 2023-10-30T12:17:11.112 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.107+0000 7eff9d3ee640 10 monclient: init 2023-10-30T12:17:11.113 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:11.113 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T12:17:11.113 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff980b4560 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:11.113 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7eff98149b50 0x7eff98143130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:11.113 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 10 monclient(hunting): picked mon.noname-c con 0x7eff98149b50 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:11.114 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff98143670 addr v1:172.21.15.148:6789/0 2023-10-30T12:17:11.114 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980b3b10 con 0x7eff98143670 2023-10-30T12:17:11.114 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980aa1b0 con 0x7eff980b4560 2023-10-30T12:17:11.114 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:11.114 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff98067160 con 0x7eff98149b50 2023-10-30T12:17:11.115 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:11.115 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.108+0000 7eff9d3ee640 10 monclient(hunting): authenticate will time out at 1523.313159s 2023-10-30T12:17:11.115 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.112+0000 7eff977fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7eff98143670 0x7eff98150790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37066/0 (socket says 172.21.15.148:37066) 2023-10-30T12:17:11.115 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.112+0000 7eff977fe640 1 -- 172.21.15.148:0/518448594 learned_addr learned my addr 172.21.15.148:0/518448594 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:17:11.142 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.141+0000 7eff95ffb640 1 -- 172.21.15.148:0/518448594 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177671195 0 0) 0x7eff90001490 con 0x7eff98143670 2023-10-30T12:17:11.142 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.141+0000 7eff95ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:17:11.143 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.141+0000 7eff95ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:11.143 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.141+0000 7eff95ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:11.143 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.141+0000 7eff95ffb640 10 monclient(hunting): my global_id is 4170 2023-10-30T12:17:11.143 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.141+0000 7eff95ffb640 1 -- 172.21.15.148:0/518448594 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff80003520 con 0x7eff98143670 2023-10-30T12:17:11.158 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.156+0000 7eff967fc640 1 --2- 172.21.15.148:0/518448594 >> v2:172.21.15.150:3300/0 conn(0x7eff98149b50 0x7eff98143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:11.158 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.157+0000 7eff967fc640 10 monclient(hunting): get_auth_request con 0x7eff98149b50 auth_method 0 2023-10-30T12:17:11.159 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.157+0000 7eff967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:11.159 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.157+0000 7eff967fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:11.159 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.157+0000 7eff967fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:11.160 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.158+0000 7eff967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:11.160 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.158+0000 7eff967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:11.160 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.158+0000 7eff967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:11.172 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.168+0000 7eff967fc640 10 monclient(hunting): handle_auth_done global_id 4177 payload 274 2023-10-30T12:17:11.173 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.168+0000 7eff967fc640 1 -- 172.21.15.148:0/518448594 >> v1:172.21.15.150:6789/0 conn(0x7eff980b4560 legacy=0x7eff98149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:11.173 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.168+0000 7eff967fc640 1 -- 172.21.15.148:0/518448594 >> v1:172.21.15.148:6789/0 conn(0x7eff98143670 legacy=0x7eff98150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:11.173 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.168+0000 7eff967fc640 10 monclient: _finish_hunting 0 2023-10-30T12:17:11.173 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.168+0000 7eff967fc640 1 monclient: found mon.noname-c 2023-10-30T12:17:11.173 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.168+0000 7eff967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:11.174 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.168+0000 7eff967fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:17:11.174 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.168+0000 7eff967fc640 1 -- 172.21.15.148:0/518448594 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff980aa490 con 0x7eff98149b50 2023-10-30T12:17:11.176 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.174+0000 7eff967fc640 1 --2- 172.21.15.148:0/518448594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff98149b50 0x7eff98143130 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097a0 tx=0x7eff8c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b107c59a269a28 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:11.180 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.179+0000 7eff95ffb640 1 -- 172.21.15.148:0/518448594 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c030860 con 0x7eff98149b50 2023-10-30T12:17:11.181 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.179+0000 7eff95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:11.181 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.179+0000 7eff95ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:11.181 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.179+0000 7eff95ffb640 10 monclient: dump: 2023-10-30T12:17:11.181 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:11.181 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:11.182 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:11.182 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:11.182 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:11.182 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:11.182 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:11.183 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:11.183 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:11.183 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:11.183 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.179+0000 7eff95ffb640 10 monclient: _finish_auth 0 2023-10-30T12:17:11.183 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.179+0000 7eff95ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:17:11.184 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.179+0000 7eff95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:11.184 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.182+0000 7eff9d3ee640 5 monclient: authenticate success, global_id 4177 2023-10-30T12:17:11.184 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.182+0000 7eff9d3ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:11.198 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.196+0000 7eff95ffb640 1 -- 172.21.15.148:0/518448594 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c0309d0 con 0x7eff98149b50 2023-10-30T12:17:11.199 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.196+0000 7eff95ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:11.199 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.196+0000 7eff95ffb640 1 -- 172.21.15.148:0/518448594 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c030d50 con 0x7eff98149b50 2023-10-30T12:17:11.199 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.196+0000 7eff95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:11.199 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.196+0000 7eff95ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:11.199 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.196+0000 7eff95ffb640 10 monclient: dump: 2023-10-30T12:17:11.200 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:11.200 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:11.200 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:11.200 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:11.201 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:11.201 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:11.201 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:11.201 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:11.201 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:11.202 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:11.202 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.196+0000 7eff9d3ee640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:11.202 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.196+0000 7eff9d3ee640 4 set_mon_vals no callback set 2023-10-30T12:17:11.202 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.197+0000 7eff9d3ee640 10 monclient: shutdown 2023-10-30T12:17:11.202 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.197+0000 7eff9d3ee640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:11.203 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.197+0000 7eff9d3ee640 1 -- 172.21.15.148:0/518448594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff98149b50 msgr2=0x7eff98143130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:11.203 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.197+0000 7eff9d3ee640 1 --2- 172.21.15.148:0/518448594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff98149b50 0x7eff98143130 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097a0 tx=0x7eff8c02b9a0 comp rx=0 tx=0).stop 2023-10-30T12:17:11.203 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.197+0000 7eff9d3ee640 1 -- 172.21.15.148:0/518448594 shutdown_connections 2023-10-30T12:17:11.203 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.197+0000 7eff9d3ee640 1 --2- 172.21.15.148:0/518448594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff98149b50 0x7eff98143130 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:11.203 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.197+0000 7eff9d3ee640 1 -- 172.21.15.148:0/518448594 >> 172.21.15.148:0/518448594 conn(0x7eff98060bf0 msgr2=0x7eff98061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:11.204 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.197+0000 7eff9d3ee640 1 -- 172.21.15.148:0/518448594 shutdown_connections 2023-10-30T12:17:11.204 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.197+0000 7eff9d3ee640 1 -- 172.21.15.148:0/518448594 wait complete. 2023-10-30T12:17:11.206 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.198+0000 7eff9d3ee640 10 monclient: build_initial_monmap 2023-10-30T12:17:11.206 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.198+0000 7eff9d3ee640 10 monclient: monmap: 2023-10-30T12:17:11.206 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:11.206 INFO:teuthology.misc.health.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:11.207 INFO:teuthology.misc.health.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:11.207 INFO:teuthology.misc.health.smithi148.stderr:created 0.000000 2023-10-30T12:17:11.207 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:11.207 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:11.207 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:11.208 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:11.208 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:11.208 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:11.208 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.198+0000 7eff9d3ee640 1 librados: starting msgr at 2023-10-30T12:17:11.208 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.198+0000 7eff9d3ee640 1 librados: starting objecter 2023-10-30T12:17:11.208 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.199+0000 7eff9d3ee640 1 Processor -- start 2023-10-30T12:17:11.209 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.199+0000 7eff9d3ee640 1 -- start start 2023-10-30T12:17:11.209 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.199+0000 7eff9d3ee640 1 librados: setting wanted keys 2023-10-30T12:17:11.209 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.199+0000 7eff9d3ee640 1 librados: calling monclient init 2023-10-30T12:17:11.209 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.199+0000 7eff9d3ee640 10 monclient: init 2023-10-30T12:17:11.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:11.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:17:11.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff980b4560 0x7eff98144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:11.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 10 monclient(hunting): picked mon.noname-b con 0x7eff980b4560 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:11.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff98143670 0x7eff98145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:11.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 10 monclient(hunting): picked mon.noname-a con 0x7eff98143670 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:11.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7eff98149b50 0x7eff980b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:11.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 10 monclient(hunting): picked mon.noname-c con 0x7eff98149b50 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:11.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:11.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff980738f0 con 0x7eff98143670 2023-10-30T12:17:11.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:11.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff980abc00 con 0x7eff980b4560 2023-10-30T12:17:11.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:11.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff98145840 con 0x7eff98149b50 2023-10-30T12:17:11.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:11.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.200+0000 7eff9d3ee640 10 monclient(hunting): authenticate will time out at 1523.405505s 2023-10-30T12:17:11.215 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.206+0000 7eff967fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff98143670 0x7eff98145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:11.215 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.207+0000 7eff967fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff98143670 0x7eff98145300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38214/0 (socket says 172.21.15.148:38214) 2023-10-30T12:17:11.215 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.207+0000 7eff967fc640 1 -- 172.21.15.148:0/2336597045 learned_addr learned my addr 172.21.15.148:0/2336597045 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:11.216 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.207+0000 7eff967fc640 10 monclient(hunting): get_auth_request con 0x7eff98143670 auth_method 0 2023-10-30T12:17:11.216 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.207+0000 7eff967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:11.216 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.207+0000 7eff967fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:11.216 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.207+0000 7eff967fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:11.217 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.207+0000 7eff967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:11.217 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.207+0000 7eff967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:11.217 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.207+0000 7eff967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:11.217 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.211+0000 7eff967fc640 10 monclient(hunting): handle_auth_done global_id 4173 payload 824 2023-10-30T12:17:11.217 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.211+0000 7eff967fc640 1 -- 172.21.15.148:0/2336597045 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7eff98149b50 msgr2=0x7eff980b09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:11.217 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.211+0000 7eff967fc640 1 --2- 172.21.15.148:0/2336597045 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7eff98149b50 0x7eff980b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:11.218 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.211+0000 7eff967fc640 1 -- 172.21.15.148:0/2336597045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff980b4560 msgr2=0x7eff98144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:11.218 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.211+0000 7eff967fc640 1 --2- 172.21.15.148:0/2336597045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff980b4560 0x7eff98144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:11.218 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.211+0000 7eff967fc640 10 monclient: _finish_hunting 0 2023-10-30T12:17:11.218 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.211+0000 7eff967fc640 1 monclient: found mon.noname-a 2023-10-30T12:17:11.218 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.211+0000 7eff967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:11.219 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.211+0000 7eff967fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:17:11.219 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.211+0000 7eff967fc640 1 -- 172.21.15.148:0/2336597045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff980ada80 con 0x7eff98143670 2023-10-30T12:17:11.219 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.217+0000 7eff967fc640 1 --2- 172.21.15.148:0/2336597045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff98143670 0x7eff98145300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0098c0 tx=0x7eff8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:11.220 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.217+0000 7eff7f7fe640 10 client.?.objecter ms_handle_connect 0x7eff98143670 2023-10-30T12:17:11.220 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.217+0000 7eff7f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:11.229 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 1 -- 172.21.15.148:0/2336597045 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c003610 con 0x7eff98143670 2023-10-30T12:17:11.230 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:11.230 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:11.230 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 10 monclient: dump: 2023-10-30T12:17:11.230 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:11.230 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:11.231 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:11.231 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:11.231 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:11.231 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:11.232 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:11.232 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:11.232 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:11.232 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:11.232 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:11.233 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:11.233 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:11.233 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 1 -- 172.21.15.148:0/2336597045 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c003780 con 0x7eff98143670 2023-10-30T12:17:11.233 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:11.233 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 1 -- 172.21.15.148:0/2336597045 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c003b00 con 0x7eff98143670 2023-10-30T12:17:11.234 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:11.234 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:11.234 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff7f7fe640 10 monclient: dump: 2023-10-30T12:17:11.234 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:11.234 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:11.235 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:11.235 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:11.235 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:11.235 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:11.235 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:11.236 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:11.236 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:11.236 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:11.237 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff957fa640 4 set_mon_vals no callback set 2023-10-30T12:17:11.237 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.226+0000 7eff9d3ee640 5 monclient: authenticate success, global_id 4173 2023-10-30T12:17:11.237 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.227+0000 7eff9d3ee640 10 monclient: _renew_subs 2023-10-30T12:17:11.237 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.227+0000 7eff9d3ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:11.237 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.227+0000 7eff9d3ee640 1 -- 172.21.15.148:0/2336597045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff980b0ee0 con 0x7eff98143670 2023-10-30T12:17:11.238 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.227+0000 7eff9d3ee640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:11.238 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.227+0000 7eff9d3ee640 10 monclient: _renew_subs 2023-10-30T12:17:11.238 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.227+0000 7eff9d3ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:11.238 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.227+0000 7eff9d3ee640 1 -- 172.21.15.148:0/2336597045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff980b12f0 con 0x7eff98143670 2023-10-30T12:17:11.238 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.227+0000 7eff9d3ee640 1 librados: init done 2023-10-30T12:17:11.239 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.229+0000 7eff9d3ee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:11.239 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.229+0000 7eff9d3ee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:11.240 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.229+0000 7eff9d3ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:11.240 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.229+0000 7eff9d3ee640 1 -- 172.21.15.148:0/2336597045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff98146aa0 con 0x7eff98143670 2023-10-30T12:17:11.247 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.246+0000 7eff7f7fe640 1 -- 172.21.15.148:0/2336597045 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 3) v1 ==== 44658+0+0 (secure 0 0 0) 0x7eff8c003c70 con 0x7eff98143670 2023-10-30T12:17:11.248 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.246+0000 7eff7f7fe640 1 --2- 172.21.15.148:0/2336597045 >> v2:172.21.15.148:6800/3205090179 conn(0x7eff84039100 0x7eff8403b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:11.248 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.247+0000 7eff96ffd640 1 --2- 172.21.15.148:0/2336597045 >> v2:172.21.15.148:6800/3205090179 conn(0x7eff84039100 0x7eff8403b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:11.249 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.247+0000 7eff96ffd640 10 monclient: get_auth_request con 0x7eff84039100 auth_method 0 2023-10-30T12:17:11.249 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.247+0000 7eff96ffd640 1 --2- 172.21.15.148:0/2336597045 >> v2:172.21.15.148:6800/3205090179 conn(0x7eff84039100 0x7eff8403b5c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7eff88000f60 tx=0x7eff880079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:11.249 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.247+0000 7eff7f7fe640 10 client.4173.objecter ms_handle_connect 0x7eff84039100 2023-10-30T12:17:11.250 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:11.249+0000 7f4e338fd640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-30T12:17:11.251 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.249+0000 7eff7f7fe640 1 -- 172.21.15.148:0/2336597045 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7eff8c042c20 con 0x7eff98143670 2023-10-30T12:17:11.251 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.249+0000 7eff7f7fe640 10 client.4173.objecter ms_dispatch 0x7eff98000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:17:11.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.249+0000 7eff7f7fe640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:17:11.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.249+0000 7eff7f7fe640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:17:11.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.249+0000 7eff7f7fe640 20 client.4173.objecter dump_active .. 0 homeless 2023-10-30T12:17:11.395 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T12:17:11.394+0000 7f4e338fd640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T12:17:11.704 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.702+0000 7eff7f7fe640 1 -- 172.21.15.148:0/2336597045 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7eff8c041370 con 0x7eff98143670 2023-10-30T12:17:11.704 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.702+0000 7eff7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:11.704 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:11.702+0000 7eff7f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:12.063 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.061+0000 7eff9d3ee640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:12.063 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.062+0000 7eff9d3ee640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:12.064 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.062+0000 7eff9d3ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:12.064 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.062+0000 7eff9d3ee640 1 -- 172.21.15.148:0/2336597045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff980ada80 con 0x7eff98143670 2023-10-30T12:17:12.141 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.140+0000 7eff7f7fe640 1 -- 172.21.15.148:0/2336597045 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7eff8c02fcb0 con 0x7eff98143670 2023-10-30T12:17:12.142 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.140+0000 7eff7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:12.142 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.140+0000 7eff7f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:12.142 INFO:teuthology.misc.health.smithi148.stdout: 2023-10-30T12:17:12.143 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":10,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:17:09.070671+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:17:12.145 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.142+0000 7eff9d3ee640 10 librados: watch_flush enter 2023-10-30T12:17:12.146 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 10 librados: watch_flush exit 2023-10-30T12:17:12.146 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 20 client.4173.objecter shutdown clearing up homeless session... 2023-10-30T12:17:12.146 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 10 client.4173.objecter successfully canceled tick 2023-10-30T12:17:12.146 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 -- 172.21.15.148:0/2336597045 >> v2:172.21.15.148:6800/3205090179 conn(0x7eff84039100 msgr2=0x7eff8403b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:12.146 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 --2- 172.21.15.148:0/2336597045 >> v2:172.21.15.148:6800/3205090179 conn(0x7eff84039100 0x7eff8403b5c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7eff88000f60 tx=0x7eff880079b0 comp rx=0 tx=0).stop 2023-10-30T12:17:12.147 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 10 monclient: shutdown 2023-10-30T12:17:12.147 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:12.147 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 -- 172.21.15.148:0/2336597045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff98143670 msgr2=0x7eff98145300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:12.147 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 --2- 172.21.15.148:0/2336597045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff98143670 0x7eff98145300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0098c0 tx=0x7eff8c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:17:12.147 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 -- 172.21.15.148:0/2336597045 shutdown_connections 2023-10-30T12:17:12.148 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 --2- 172.21.15.148:0/2336597045 >> v2:172.21.15.148:6800/3205090179 conn(0x7eff84039100 0x7eff8403b5c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:12.148 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 --2- 172.21.15.148:0/2336597045 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7eff98149b50 0x7eff980b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:12.148 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 --2- 172.21.15.148:0/2336597045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff98143670 0x7eff98145300 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:12.148 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 --2- 172.21.15.148:0/2336597045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff980b4560 0x7eff98144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:12.148 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 -- 172.21.15.148:0/2336597045 >> 172.21.15.148:0/2336597045 conn(0x7eff98060bf0 msgr2=0x7eff980b6880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:12.149 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 -- 172.21.15.148:0/2336597045 shutdown_connections 2023-10-30T12:17:12.149 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 -- 172.21.15.148:0/2336597045 wait complete. 2023-10-30T12:17:12.149 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:12.143+0000 7eff9d3ee640 1 librados: shutdown 2023-10-30T12:17:12.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:17:17.507 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:08.050 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:17.546 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:08.091 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:17.572 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:08.114 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:17.611 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:08.164 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:17.739 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:08.295 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:17.805 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:08.360 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:17.825 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:08.387 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:17.897 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:08.442 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:17.902 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:08.464 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:17.955 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:08.500 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.037 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:08.626 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.070 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:08.644 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.101 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:08.691 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.117 INFO:tasks.ceph.osd.6.smithi150.stderr:2023-10-30T12:17:18.103+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:17:18.130 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:08.704 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.155 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:08.736 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.181 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T12:17:18.173+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:17:18.214 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:08.794 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.384 INFO:tasks.ceph.osd.4.smithi150.stderr:2023-10-30T12:17:18.376+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:17:18.446 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:17:18.439+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:17:18.541 INFO:tasks.ceph.osd.7.smithi150.stderr:2023-10-30T12:17:18.532+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:17:18.562 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:09.105 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.607 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:09.160 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.700 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:17:18.691+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:17:18.706 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T12:17:18.699+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:17:18.819 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T12:17:18.809+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:17:18.839 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:09.395 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.919 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:09.462 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.961 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:09.523 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.962 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:09.515 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:18.967 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:17:18.978 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:09.523 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.061 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.060+0000 7f2788f16640 10 monclient: get_monmap_and_config 2023-10-30T12:17:19.062 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.060+0000 7f2788f16640 10 monclient: build_initial_monmap 2023-10-30T12:17:19.062 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.060+0000 7f2788f16640 10 monclient: monmap: 2023-10-30T12:17:19.062 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:19.063 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:19.063 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:17:19.061498+0000 2023-10-30T12:17:19.063 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:17:19.061498+0000 2023-10-30T12:17:19.063 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:19.063 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:19.064 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:19.064 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:19.064 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:19.064 INFO:teuthology.misc.health.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:19.064 INFO:teuthology.misc.health.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:19.065 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:19.068 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 1 Processor -- start 2023-10-30T12:17:19.068 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 1 -- start start 2023-10-30T12:17:19.068 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 10 monclient: init 2023-10-30T12:17:19.068 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:19.068 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-10-30T12:17:19.069 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27840a9110 0x7f27840b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:19.069 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 10 monclient(hunting): picked mon.noname-c con 0x7f27840a9110 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:19.069 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f27840a94f0 0x7f2784145880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:19.069 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 10 monclient(hunting): picked mon.noname-b con 0x7f27840a94f0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:19.069 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2784145dc0 addr v1:172.21.15.148:6789/0 2023-10-30T12:17:19.070 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27840aa800 con 0x7f2784145dc0 2023-10-30T12:17:19.070 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:19.070 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27840671a0 con 0x7f27840a9110 2023-10-30T12:17:19.070 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:19.071 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27840aaae0 con 0x7f27840a94f0 2023-10-30T12:17:19.071 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:19.071 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.061+0000 7f2788f16640 10 monclient(hunting): authenticate will time out at 1531.266880s 2023-10-30T12:17:19.071 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.062+0000 7f278259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27840a9110 0x7f27840b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:19.072 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.062+0000 7f2781d9b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f27840a94f0 0x7f2784145880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:19.072 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.062+0000 7f2782d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2784145dc0 0x7f278413ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37080/0 (socket says 172.21.15.148:37080) 2023-10-30T12:17:19.072 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.062+0000 7f2782d9d640 1 -- 172.21.15.148:0/2637426635 learned_addr learned my addr 172.21.15.148:0/2637426635 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:17:19.072 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.062+0000 7f278259c640 10 monclient(hunting): get_auth_request con 0x7f27840a9110 auth_method 0 2023-10-30T12:17:19.073 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.062+0000 7f278259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:19.073 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.062+0000 7f278259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:19.073 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.062+0000 7f278259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:19.073 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.063+0000 7f2781d9b640 10 monclient(hunting): get_auth_request con 0x7f27840a94f0 auth_method 0 2023-10-30T12:17:19.073 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.063+0000 7f2781d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:19.074 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.063+0000 7f2781d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:19.074 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.063+0000 7f2781d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:19.074 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.063+0000 7f278259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:19.075 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.063+0000 7f278259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:19.075 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.063+0000 7f278259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:19.075 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.063+0000 7f2781d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:19.075 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.063+0000 7f2781d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:19.076 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.063+0000 7f2781d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:19.076 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.065+0000 7f278259c640 10 monclient(hunting): handle_auth_done global_id 4201 payload 274 2023-10-30T12:17:19.076 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.065+0000 7f278259c640 1 -- 172.21.15.148:0/2637426635 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f27840a94f0 msgr2=0x7f2784145880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:19.076 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.065+0000 7f278259c640 1 --2- 172.21.15.148:0/2637426635 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f27840a94f0 0x7f2784145880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:19.077 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.065+0000 7f278259c640 1 -- 172.21.15.148:0/2637426635 >> v1:172.21.15.148:6789/0 conn(0x7f2784145dc0 legacy=0x7f278413ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:19.077 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.065+0000 7f278259c640 10 monclient: _finish_hunting 0 2023-10-30T12:17:19.077 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.065+0000 7f278259c640 1 monclient: found mon.noname-c 2023-10-30T12:17:19.077 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.065+0000 7f278259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:19.077 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.065+0000 7f278259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:17:19.078 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.065+0000 7f278259c640 1 -- 172.21.15.148:0/2637426635 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2784145080 con 0x7f27840a9110 2023-10-30T12:17:19.083 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.067+0000 7f278259c640 1 --2- 172.21.15.148:0/2637426635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27840a9110 0x7f27840b1da0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f27740097a0 tx=0x7f277402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56d0d85ef718881c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:19.083 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 1 -- 172.21.15.148:0/2637426635 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774031700 con 0x7f27840a9110 2023-10-30T12:17:19.083 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:19.084 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:19.084 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 10 monclient: dump: 2023-10-30T12:17:19.084 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:19.084 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:19.084 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:19.085 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:19.085 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:19.085 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:19.085 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:19.086 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:19.086 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:19.086 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:19.086 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 10 monclient: _finish_auth 0 2023-10-30T12:17:19.086 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 10 monclient: _check_auth_tickets 2023-10-30T12:17:19.087 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:19.087 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 1 -- 172.21.15.148:0/2637426635 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2774031870 con 0x7f27840a9110 2023-10-30T12:17:19.087 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f2788f16640 5 monclient: authenticate success, global_id 4201 2023-10-30T12:17:19.087 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:19.088 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 1 -- 172.21.15.148:0/2637426635 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774031c10 con 0x7f27840a9110 2023-10-30T12:17:19.088 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f2788f16640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:19.088 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.071+0000 7f2788f16640 4 set_mon_vals no callback set 2023-10-30T12:17:19.088 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.070+0000 7f278159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:19.088 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.071+0000 7f278159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:19.089 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.071+0000 7f278159a640 10 monclient: dump: 2023-10-30T12:17:19.089 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:19.089 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:19.089 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:19.090 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:19.090 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:19.090 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:19.090 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:19.091 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:19.091 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:19.091 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:19.091 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.071+0000 7f2788f16640 10 monclient: shutdown 2023-10-30T12:17:19.091 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.071+0000 7f2788f16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:19.092 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.071+0000 7f2788f16640 1 -- 172.21.15.148:0/2637426635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27840a9110 msgr2=0x7f27840b1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:19.092 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.071+0000 7f2788f16640 1 --2- 172.21.15.148:0/2637426635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27840a9110 0x7f27840b1da0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f27740097a0 tx=0x7f277402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:17:19.092 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.072+0000 7f2788f16640 1 -- 172.21.15.148:0/2637426635 shutdown_connections 2023-10-30T12:17:19.092 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.072+0000 7f2788f16640 1 --2- 172.21.15.148:0/2637426635 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f27840a94f0 0x7f2784145880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:19.092 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.072+0000 7f2788f16640 1 --2- 172.21.15.148:0/2637426635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27840a9110 0x7f27840b1da0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:19.092 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.072+0000 7f2788f16640 1 -- 172.21.15.148:0/2637426635 >> 172.21.15.148:0/2637426635 conn(0x7f27840b7420 msgr2=0x7f27840b7830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:19.093 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.072+0000 7f2788f16640 1 -- 172.21.15.148:0/2637426635 shutdown_connections 2023-10-30T12:17:19.093 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.072+0000 7f2788f16640 1 -- 172.21.15.148:0/2637426635 wait complete. 2023-10-30T12:17:19.093 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.073+0000 7f2788f16640 10 monclient: build_initial_monmap 2023-10-30T12:17:19.093 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.073+0000 7f2788f16640 10 monclient: monmap: 2023-10-30T12:17:19.093 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:19.094 INFO:teuthology.misc.health.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:19.094 INFO:teuthology.misc.health.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:19.094 INFO:teuthology.misc.health.smithi148.stderr:created 0.000000 2023-10-30T12:17:19.094 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:19.094 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:19.095 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:19.095 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:19.095 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:19.095 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:19.095 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.073+0000 7f2788f16640 1 librados: starting msgr at 2023-10-30T12:17:19.096 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.073+0000 7f2788f16640 1 librados: starting objecter 2023-10-30T12:17:19.096 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.073+0000 7f2788f16640 1 Processor -- start 2023-10-30T12:17:19.096 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.073+0000 7f2788f16640 1 -- start start 2023-10-30T12:17:19.096 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.073+0000 7f2788f16640 1 librados: setting wanted keys 2023-10-30T12:17:19.096 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.073+0000 7f2788f16640 1 librados: calling monclient init 2023-10-30T12:17:19.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.073+0000 7f2788f16640 10 monclient: init 2023-10-30T12:17:19.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:19.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:17:19.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27840a9110 0x7f2784140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:19.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 10 monclient(hunting): picked mon.noname-a con 0x7f27840a9110 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:19.098 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f27840a94f0 0x7f2784140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:19.098 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 10 monclient(hunting): picked mon.noname-c con 0x7f27840a94f0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:19.098 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278413c8b0 0x7f27841da5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:19.098 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 10 monclient(hunting): picked mon.noname-b con 0x7f278413c8b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:19.098 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:19.098 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27840b1a50 con 0x7f27840a9110 2023-10-30T12:17:19.099 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:19.099 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2784141a30 con 0x7f278413c8b0 2023-10-30T12:17:19.099 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:19.099 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27841da160 con 0x7f27840a94f0 2023-10-30T12:17:19.099 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:19.100 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.074+0000 7f2788f16640 10 monclient(hunting): authenticate will time out at 1531.279834s 2023-10-30T12:17:19.100 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.075+0000 7f278259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27840a9110 0x7f2784140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:19.100 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.075+0000 7f2781d9b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f27840a94f0 0x7f2784140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:19.100 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.075+0000 7f2782d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278413c8b0 0x7f27841da5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:19.100 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2782d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278413c8b0 0x7f27841da5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:43124/0 (socket says 172.21.15.148:43124) 2023-10-30T12:17:19.101 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2782d9d640 1 -- 172.21.15.148:0/4280533532 learned_addr learned my addr 172.21.15.148:0/4280533532 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:19.101 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2782d9d640 10 monclient(hunting): get_auth_request con 0x7f278413c8b0 auth_method 0 2023-10-30T12:17:19.101 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2782d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:19.101 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2782d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:19.101 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2782d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:19.102 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f278259c640 10 monclient(hunting): get_auth_request con 0x7f27840a9110 auth_method 0 2023-10-30T12:17:19.102 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f278259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:19.102 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f278259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:19.102 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f278259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:19.102 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2781d9b640 10 monclient(hunting): get_auth_request con 0x7f27840a94f0 auth_method 0 2023-10-30T12:17:19.103 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2781d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:19.103 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2781d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:19.103 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2781d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:19.103 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2782d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:19.104 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2782d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:19.104 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f2782d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:19.104 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f278259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:19.104 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f278259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:19.104 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.076+0000 7f278259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:19.105 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.077+0000 7f2781d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:19.105 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.077+0000 7f2781d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:19.105 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.077+0000 7f2781d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:19.105 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.078+0000 7f2782d9d640 10 monclient(hunting): handle_auth_done global_id 4204 payload 824 2023-10-30T12:17:19.106 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.078+0000 7f2782d9d640 1 -- 172.21.15.148:0/4280533532 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f27840a94f0 msgr2=0x7f2784140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:19.106 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.078+0000 7f2782d9d640 1 --2- 172.21.15.148:0/4280533532 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f27840a94f0 0x7f2784140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:19.106 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.078+0000 7f2782d9d640 1 -- 172.21.15.148:0/4280533532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27840a9110 msgr2=0x7f2784140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:19.106 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.078+0000 7f2782d9d640 1 --2- 172.21.15.148:0/4280533532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27840a9110 0x7f2784140dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:19.107 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.079+0000 7f2782d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:17:19.107 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.079+0000 7f2782d9d640 1 monclient: found mon.noname-b 2023-10-30T12:17:19.107 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.079+0000 7f2782d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:19.107 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.079+0000 7f2782d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:19.108 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.079+0000 7f2782d9d640 1 -- 172.21.15.148:0/4280533532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27841408c0 con 0x7f278413c8b0 2023-10-30T12:17:19.108 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.081+0000 7f2782d9d640 1 --2- 172.21.15.148:0/4280533532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278413c8b0 0x7f27841da5f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f277c00cad0 tx=0x7f277c00cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:19.108 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.081+0000 7f2772ffd640 10 client.?.objecter ms_handle_connect 0x7f278413c8b0 2023-10-30T12:17:19.108 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.081+0000 7f2772ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:19.111 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 1 -- 172.21.15.148:0/4280533532 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c00abd0 con 0x7f278413c8b0 2023-10-30T12:17:19.112 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:19.112 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:19.112 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 10 monclient: dump: 2023-10-30T12:17:19.112 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:19.113 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:19.113 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:19.113 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:19.113 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:19.113 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:19.114 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:19.114 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:19.114 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:19.114 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:19.115 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 10 monclient: _finish_auth 0 2023-10-30T12:17:19.115 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:19.115 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:19.115 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 1 -- 172.21.15.148:0/4280533532 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277c00e040 con 0x7f278413c8b0 2023-10-30T12:17:19.116 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2788f16640 5 monclient: authenticate success, global_id 4204 2023-10-30T12:17:19.116 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2788f16640 10 monclient: _renew_subs 2023-10-30T12:17:19.116 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2788f16640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:19.116 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2788f16640 1 -- 172.21.15.148:0/4280533532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278413e7a0 con 0x7f278413c8b0 2023-10-30T12:17:19.116 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:19.117 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 1 -- 172.21.15.148:0/4280533532 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c012400 con 0x7f278413c8b0 2023-10-30T12:17:19.117 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:19.117 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:19.117 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2772ffd640 10 monclient: dump: 2023-10-30T12:17:19.118 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:19.118 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:19.118 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:19.118 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:19.119 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:19.119 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:19.119 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:19.119 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:19.120 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:19.120 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:19.120 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2780d99640 4 set_mon_vals no callback set 2023-10-30T12:17:19.120 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2788f16640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:19.120 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2788f16640 10 monclient: _renew_subs 2023-10-30T12:17:19.121 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2788f16640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:19.121 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2788f16640 1 -- 172.21.15.148:0/4280533532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27841dad10 con 0x7f278413c8b0 2023-10-30T12:17:19.121 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.084+0000 7f2788f16640 1 librados: init done 2023-10-30T12:17:19.121 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.085+0000 7f2788f16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:19.121 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.085+0000 7f2788f16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:19.122 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.085+0000 7f2788f16640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:19.122 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.085+0000 7f2788f16640 1 -- 172.21.15.148:0/4280533532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27841da310 con 0x7f278413c8b0 2023-10-30T12:17:19.122 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.102+0000 7f2772ffd640 1 -- 172.21.15.148:0/4280533532 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7f277c007070 con 0x7f278413c8b0 2023-10-30T12:17:19.122 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.103+0000 7f2772ffd640 1 --2- 172.21.15.148:0/4280533532 >> v2:172.21.15.148:6800/3205090179 conn(0x7f276406c730 0x7f276406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:19.122 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.103+0000 7f278259c640 1 --2- 172.21.15.148:0/4280533532 >> v2:172.21.15.148:6800/3205090179 conn(0x7f276406c730 0x7f276406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:19.123 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.103+0000 7f278259c640 10 monclient: get_auth_request con 0x7f276406c730 auth_method 0 2023-10-30T12:17:19.123 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.103+0000 7f2772ffd640 10 client.4204.objecter ms_handle_connect 0x7f276406c730 2023-10-30T12:17:19.123 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.103+0000 7f278259c640 1 --2- 172.21.15.148:0/4280533532 >> v2:172.21.15.148:6800/3205090179 conn(0x7f276406c730 0x7f276406ebf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2774035000 tx=0x7f2774008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:19.123 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.104+0000 7f2772ffd640 1 -- 172.21.15.148:0/4280533532 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f277c057e60 con 0x7f278413c8b0 2023-10-30T12:17:19.123 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.104+0000 7f2772ffd640 10 client.4204.objecter ms_dispatch 0x7f2784000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:17:19.124 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.104+0000 7f2772ffd640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:17:19.124 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.104+0000 7f2772ffd640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:17:19.124 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.104+0000 7f2772ffd640 20 client.4204.objecter dump_active .. 0 homeless 2023-10-30T12:17:19.153 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:09.742 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.153 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:09.727 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.196 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:09.752 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.233 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:09.776 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.268 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:09.849 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.291 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:09.844 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.318 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:09.880 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.347 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:09.892 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.353 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.352+0000 7f2772ffd640 1 -- 172.21.15.148:0/4280533532 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f277c05bf00 con 0x7f278413c8b0 2023-10-30T12:17:19.354 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.352+0000 7f2772ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:19.354 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.352+0000 7f2772ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:19.509 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:10.052 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.511 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:10.100 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.519 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:10.075 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.528 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:10.102 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.552 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.550+0000 7f2788f16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:19.552 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.550+0000 7f2788f16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:19.552 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.550+0000 7f2788f16640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:19.552 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.550+0000 7f2788f16640 1 -- 172.21.15.148:0/4280533532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f278413e7a0 con 0x7f278413c8b0 2023-10-30T12:17:19.561 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:10.115 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.567 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.565+0000 7f2772ffd640 1 -- 172.21.15.148:0/4280533532 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f277c0600b0 con 0x7f278413c8b0 2023-10-30T12:17:19.567 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.566+0000 7f2772ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:19.567 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.566+0000 7f2772ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:19.568 INFO:teuthology.misc.health.smithi148.stdout: 2023-10-30T12:17:19.568 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":10,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:17:09.070671+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:17:19.569 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.567+0000 7f2788f16640 10 librados: watch_flush enter 2023-10-30T12:17:19.570 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.567+0000 7f2788f16640 10 librados: watch_flush exit 2023-10-30T12:17:19.570 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.567+0000 7f2788f16640 20 client.4204.objecter shutdown clearing up homeless session... 2023-10-30T12:17:19.570 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.567+0000 7f2788f16640 10 client.4204.objecter successfully canceled tick 2023-10-30T12:17:19.570 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.567+0000 7f2788f16640 1 -- 172.21.15.148:0/4280533532 >> v2:172.21.15.148:6800/3205090179 conn(0x7f276406c730 msgr2=0x7f276406ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:19.571 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 --2- 172.21.15.148:0/4280533532 >> v2:172.21.15.148:6800/3205090179 conn(0x7f276406c730 0x7f276406ebf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2774035000 tx=0x7f2774008030 comp rx=0 tx=0).stop 2023-10-30T12:17:19.571 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 10 monclient: shutdown 2023-10-30T12:17:19.571 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:19.571 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 -- 172.21.15.148:0/4280533532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278413c8b0 msgr2=0x7f27841da5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:19.571 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 --2- 172.21.15.148:0/4280533532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278413c8b0 0x7f27841da5f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f277c00cad0 tx=0x7f277c00cf90 comp rx=0 tx=0).stop 2023-10-30T12:17:19.572 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 -- 172.21.15.148:0/4280533532 shutdown_connections 2023-10-30T12:17:19.572 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 --2- 172.21.15.148:0/4280533532 >> v2:172.21.15.148:6800/3205090179 conn(0x7f276406c730 0x7f276406ebf0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:19.572 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 --2- 172.21.15.148:0/4280533532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278413c8b0 0x7f27841da5f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:19.572 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 --2- 172.21.15.148:0/4280533532 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f27840a94f0 0x7f2784140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:19.572 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 --2- 172.21.15.148:0/4280533532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27840a9110 0x7f2784140dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:19.573 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 -- 172.21.15.148:0/4280533532 >> 172.21.15.148:0/4280533532 conn(0x7f27840b7420 msgr2=0x7f27840b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:19.573 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 -- 172.21.15.148:0/4280533532 shutdown_connections 2023-10-30T12:17:19.573 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 -- 172.21.15.148:0/4280533532 wait complete. 2023-10-30T12:17:19.573 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:19.568+0000 7f2788f16640 1 librados: shutdown 2023-10-30T12:17:19.579 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:17:19.621 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:10.202 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.650 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:10.212 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.669 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:10.214 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.774 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:10.317 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.788 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:10.343 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.826 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:10.415 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.837 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:10.390 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.863 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:10.437 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.921 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:10.483 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.934 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:10.479 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:19.947 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:10.527 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.053 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:10.596 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.081 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:10.637 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.096 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:10.639 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.106 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:10.696 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.130 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:10.683 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.146 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:10.720 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.199 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:10.762 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.213 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:10.758 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.240 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:10.821 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.371 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:10.927 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.397 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:10.940 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.400 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:10.990 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.415 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:10.971 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.419 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:10.972 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.439 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:11.013 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.486 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:11.049 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.503 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:11.048 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.542 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:11.123 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.689 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:11.232 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.690 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:11.279 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.710 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:11.266 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.714 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:11.267 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.732 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:11.305 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.774 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:11.337 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.793 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:11.338 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.834 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:11.414 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.976 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:11.566 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:20.985 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:11.538 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.008 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:11.564 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.026 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:11.600 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.061 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:11.624 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.083 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:11.628 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.122 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:11.702 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.202 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:11.745 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.254 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:11.797 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.266 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:11.856 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.292 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:11.865 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.327 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:11.889 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.358 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:11.903 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.387 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:11.967 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.482 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:12.035 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.483 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:12.038 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.531 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:12.120 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.537 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:12.090 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.537 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:12.093 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.560 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:12.134 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.667 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:12.247 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.792 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:12.354 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.846 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:12.408 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.938 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:12.483 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:21.993 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:12.538 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:22.019 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:12.609 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:22.074 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:12.664 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:22.075 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:12.649 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:22.130 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:12.704 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:22.146 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:12.727 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:22.200 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:12.781 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:26.385 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:17:26.500 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.499+0000 7fe63e733640 10 monclient: get_monmap_and_config 2023-10-30T12:17:26.501 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.499+0000 7fe63e733640 10 monclient: build_initial_monmap 2023-10-30T12:17:26.501 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.499+0000 7fe63e733640 10 monclient: monmap: 2023-10-30T12:17:26.501 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:26.501 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:26.502 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:17:26.500230+0000 2023-10-30T12:17:26.502 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:17:26.500230+0000 2023-10-30T12:17:26.502 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:26.502 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:26.503 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:26.503 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:26.503 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:26.503 INFO:teuthology.misc.health.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:26.504 INFO:teuthology.misc.health.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:26.504 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:26.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.500+0000 7fe63e733640 1 Processor -- start 2023-10-30T12:17:26.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.500+0000 7fe63e733640 1 -- start start 2023-10-30T12:17:26.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.500+0000 7fe63e733640 10 monclient: init 2023-10-30T12:17:26.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:26.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-10-30T12:17:26.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe6380b4590 0x7fe638144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:26.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 10 monclient(hunting): picked mon.noname-b con 0x7fe6380b4590 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:26.509 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe638145530 0x7fe63813eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:26.509 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 10 monclient(hunting): picked mon.noname-c con 0x7fe638145530 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:26.509 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe63813f1e0 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:26.509 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6380aa540 con 0x7fe63813f1e0 2023-10-30T12:17:26.510 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:26.510 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe638067160 con 0x7fe638145530 2023-10-30T12:17:26.510 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:26.511 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe638145990 con 0x7fe6380b4590 2023-10-30T12:17:26.511 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:26.511 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.501+0000 7fe63e733640 10 monclient(hunting): authenticate will time out at 1538.706109s 2023-10-30T12:17:26.511 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.502+0000 7fe63ccd0640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe63813f1e0 0x7fe6380b1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:49254/0 (socket says 172.21.15.148:49254) 2023-10-30T12:17:26.512 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.502+0000 7fe63ccd0640 1 -- 172.21.15.148:0/3656316241 learned_addr learned my addr 172.21.15.148:0/3656316241 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:17:26.512 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.502+0000 7fe6377fe640 1 --2- 172.21.15.148:0/3656316241 >> v2:172.21.15.150:3300/0 conn(0x7fe638145530 0x7fe63813eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:26.512 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.503+0000 7fe637fff640 1 --2- 172.21.15.148:0/3656316241 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe6380b4590 0x7fe638144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:26.512 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe637fff640 10 monclient(hunting): get_auth_request con 0x7fe6380b4590 auth_method 0 2023-10-30T12:17:26.513 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:26.513 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe637fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:26.513 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe637fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:26.513 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe6377fe640 10 monclient(hunting): get_auth_request con 0x7fe638145530 auth_method 0 2023-10-30T12:17:26.513 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe6377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:26.514 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe6377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:26.514 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe6377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:26.514 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:26.514 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:26.515 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.504+0000 7fe637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:26.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.506+0000 7fe637fff640 10 monclient(hunting): handle_auth_done global_id 4178 payload 274 2023-10-30T12:17:26.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.506+0000 7fe637fff640 1 -- 172.21.15.148:0/3656316241 >> v2:172.21.15.150:3300/0 conn(0x7fe638145530 msgr2=0x7fe63813eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:26.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.506+0000 7fe637fff640 1 --2- 172.21.15.148:0/3656316241 >> v2:172.21.15.150:3300/0 conn(0x7fe638145530 0x7fe63813eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:26.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.507+0000 7fe637fff640 1 -- 172.21.15.148:0/3656316241 >> v1:172.21.15.150:6789/0 conn(0x7fe63813f1e0 legacy=0x7fe6380b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:26.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.507+0000 7fe637fff640 10 monclient: _finish_hunting 0 2023-10-30T12:17:26.518 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.507+0000 7fe637fff640 1 monclient: found mon.noname-b 2023-10-30T12:17:26.518 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.507+0000 7fe637fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:26.518 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.507+0000 7fe637fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:17:26.518 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.507+0000 7fe637fff640 1 -- 172.21.15.148:0/3656316241 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6380aa7b0 con 0x7fe6380b4590 2023-10-30T12:17:26.519 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.509+0000 7fe637fff640 1 --2- 172.21.15.148:0/3656316241 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe6380b4590 0x7fe638144ff0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe6280097a0 tx=0x7fe62802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb245064ada5b959 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:26.519 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.512+0000 7fe636ffd640 1 -- 172.21.15.148:0/3656316241 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628031780 con 0x7fe6380b4590 2023-10-30T12:17:26.519 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.512+0000 7fe636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:26.519 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.512+0000 7fe636ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:26.519 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.512+0000 7fe636ffd640 10 monclient: dump: 2023-10-30T12:17:26.520 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:26.520 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:26.520 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:26.520 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:26.520 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:26.521 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:26.521 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:26.521 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:26.521 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:26.521 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:26.522 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.512+0000 7fe636ffd640 10 monclient: _finish_auth 0 2023-10-30T12:17:26.522 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.512+0000 7fe636ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:26.522 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.512+0000 7fe636ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:26.523 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.512+0000 7fe63e733640 5 monclient: authenticate success, global_id 4178 2023-10-30T12:17:26.523 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.512+0000 7fe63e733640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:26.523 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.515+0000 7fe636ffd640 1 -- 172.21.15.148:0/3656316241 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6280318f0 con 0x7fe6380b4590 2023-10-30T12:17:26.523 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.515+0000 7fe636ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:26.523 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.515+0000 7fe636ffd640 1 -- 172.21.15.148:0/3656316241 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628031c90 con 0x7fe6380b4590 2023-10-30T12:17:26.524 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.515+0000 7fe636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:26.524 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.515+0000 7fe636ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:26.524 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.515+0000 7fe636ffd640 10 monclient: dump: 2023-10-30T12:17:26.524 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:26.525 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:26.525 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:26.525 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:26.525 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:26.525 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:26.526 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:26.526 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:26.526 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:26.527 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:26.527 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.515+0000 7fe63e733640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:26.528 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.515+0000 7fe63e733640 4 set_mon_vals no callback set 2023-10-30T12:17:26.533 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.516+0000 7fe63e733640 10 monclient: shutdown 2023-10-30T12:17:26.533 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.516+0000 7fe63e733640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:26.534 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.516+0000 7fe63e733640 1 -- 172.21.15.148:0/3656316241 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe6380b4590 msgr2=0x7fe638144ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:26.534 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.516+0000 7fe63e733640 1 --2- 172.21.15.148:0/3656316241 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe6380b4590 0x7fe638144ff0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe6280097a0 tx=0x7fe62802b820 comp rx=0 tx=0).stop 2023-10-30T12:17:26.534 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.516+0000 7fe63e733640 1 -- 172.21.15.148:0/3656316241 shutdown_connections 2023-10-30T12:17:26.534 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.516+0000 7fe63e733640 1 --2- 172.21.15.148:0/3656316241 >> v2:172.21.15.150:3300/0 conn(0x7fe638145530 0x7fe63813eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:26.535 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.516+0000 7fe63e733640 1 --2- 172.21.15.148:0/3656316241 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe6380b4590 0x7fe638144ff0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:26.535 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.516+0000 7fe63e733640 1 -- 172.21.15.148:0/3656316241 >> 172.21.15.148:0/3656316241 conn(0x7fe638060bf0 msgr2=0x7fe638061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:26.535 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.516+0000 7fe63e733640 1 -- 172.21.15.148:0/3656316241 shutdown_connections 2023-10-30T12:17:26.535 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.516+0000 7fe63e733640 1 -- 172.21.15.148:0/3656316241 wait complete. 2023-10-30T12:17:26.535 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.517+0000 7fe63e733640 10 monclient: build_initial_monmap 2023-10-30T12:17:26.536 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.517+0000 7fe63e733640 10 monclient: monmap: 2023-10-30T12:17:26.536 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:26.536 INFO:teuthology.misc.health.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:26.536 INFO:teuthology.misc.health.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:26.536 INFO:teuthology.misc.health.smithi148.stderr:created 0.000000 2023-10-30T12:17:26.537 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:26.537 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:26.537 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:26.537 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:26.537 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:26.537 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:26.538 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.518+0000 7fe63e733640 1 librados: starting msgr at 2023-10-30T12:17:26.538 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.518+0000 7fe63e733640 1 librados: starting objecter 2023-10-30T12:17:26.538 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.518+0000 7fe63e733640 1 Processor -- start 2023-10-30T12:17:26.538 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.518+0000 7fe63e733640 1 -- start start 2023-10-30T12:17:26.538 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.518+0000 7fe63e733640 1 librados: setting wanted keys 2023-10-30T12:17:26.539 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.518+0000 7fe63e733640 1 librados: calling monclient init 2023-10-30T12:17:26.539 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.518+0000 7fe63e733640 10 monclient: init 2023-10-30T12:17:26.539 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.519+0000 7fe63e733640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:26.539 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.519+0000 7fe63e733640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:17:26.539 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.519+0000 7fe63e733640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6380b4590 0x7fe63813fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:26.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.519+0000 7fe63e733640 10 monclient(hunting): picked mon.noname-b con 0x7fe6380b4590 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:26.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.520+0000 7fe63e733640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe63813f1e0 0x7fe6380ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:26.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.520+0000 7fe63e733640 10 monclient(hunting): picked mon.noname-c con 0x7fe63813f1e0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:26.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.520+0000 7fe63e733640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe638145530 0x7fe638141670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:26.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.520+0000 7fe6377fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe63813f1e0 0x7fe6380ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:26.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.520+0000 7fe637fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6380b4590 0x7fe63813fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:26.541 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.520+0000 7fe63e733640 10 monclient(hunting): picked mon.noname-a con 0x7fe638145530 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:26.541 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.520+0000 7fe63e733640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:26.541 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.520+0000 7fe63e733640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380a8e60 con 0x7fe638145530 2023-10-30T12:17:26.541 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63e733640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:26.541 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63e733640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6381d66f0 con 0x7fe6380b4590 2023-10-30T12:17:26.542 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63e733640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:26.542 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63e733640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe638140130 con 0x7fe63813f1e0 2023-10-30T12:17:26.542 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63ccd0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe638145530 0x7fe638141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:26.542 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe637fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6380b4590 0x7fe63813fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:35922/0 (socket says 172.21.15.148:35922) 2023-10-30T12:17:26.542 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe637fff640 1 -- 172.21.15.148:0/559363425 learned_addr learned my addr 172.21.15.148:0/559363425 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:26.543 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63e733640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:26.543 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63e733640 10 monclient(hunting): authenticate will time out at 1538.726311s 2023-10-30T12:17:26.543 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe637fff640 10 monclient(hunting): get_auth_request con 0x7fe6380b4590 auth_method 0 2023-10-30T12:17:26.543 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:26.543 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe637fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:26.544 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe637fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:26.544 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63ccd0640 10 monclient(hunting): get_auth_request con 0x7fe638145530 auth_method 0 2023-10-30T12:17:26.544 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63ccd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:26.544 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63ccd0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:26.544 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe63ccd0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:26.544 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe6377fe640 10 monclient(hunting): get_auth_request con 0x7fe63813f1e0 auth_method 0 2023-10-30T12:17:26.545 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe6377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:26.545 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe6377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:26.545 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.521+0000 7fe6377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:26.545 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.522+0000 7fe637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:26.545 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.522+0000 7fe637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:26.546 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.522+0000 7fe637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:26.546 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.522+0000 7fe63ccd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:26.546 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.522+0000 7fe63ccd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:26.546 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.522+0000 7fe63ccd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:26.546 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.522+0000 7fe6377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:26.547 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.522+0000 7fe6377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:26.547 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.522+0000 7fe6377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:26.547 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.525+0000 7fe63ccd0640 10 monclient(hunting): handle_auth_done global_id 4215 payload 824 2023-10-30T12:17:26.547 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.525+0000 7fe63ccd0640 1 -- 172.21.15.148:0/559363425 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe63813f1e0 msgr2=0x7fe6380ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:26.547 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.525+0000 7fe63ccd0640 1 --2- 172.21.15.148:0/559363425 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe63813f1e0 0x7fe6380ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:26.548 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.525+0000 7fe63ccd0640 1 -- 172.21.15.148:0/559363425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6380b4590 msgr2=0x7fe63813fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:26.548 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.525+0000 7fe63ccd0640 1 --2- 172.21.15.148:0/559363425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6380b4590 0x7fe63813fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:26.548 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.525+0000 7fe63ccd0640 10 monclient: _finish_hunting 0 2023-10-30T12:17:26.548 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.525+0000 7fe63ccd0640 1 monclient: found mon.noname-a 2023-10-30T12:17:26.548 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.525+0000 7fe63ccd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:26.549 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.525+0000 7fe63ccd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:17:26.549 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.525+0000 7fe63ccd0640 1 -- 172.21.15.148:0/559363425 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6380ad060 con 0x7fe638145530 2023-10-30T12:17:26.549 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.529+0000 7fe63ccd0640 1 --2- 172.21.15.148:0/559363425 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe638145530 0x7fe638141670 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe63000caa0 tx=0x7fe63000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:26.549 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.529+0000 7fe634ff9640 10 client.?.objecter ms_handle_connect 0x7fe638145530 2023-10-30T12:17:26.549 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.529+0000 7fe634ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:26.550 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 1 -- 172.21.15.148:0/559363425 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63000ab40 con 0x7fe638145530 2023-10-30T12:17:26.551 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:26.551 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:26.551 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 10 monclient: dump: 2023-10-30T12:17:26.551 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:26.552 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:26.552 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:26.552 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:26.552 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:26.552 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:26.552 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:26.553 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:26.553 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:26.553 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:26.553 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 10 monclient: _finish_auth 0 2023-10-30T12:17:26.553 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:17:26.554 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:26.554 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 1 -- 172.21.15.148:0/559363425 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63000e040 con 0x7fe638145530 2023-10-30T12:17:26.554 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe63e733640 5 monclient: authenticate success, global_id 4215 2023-10-30T12:17:26.554 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:26.555 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 1 -- 172.21.15.148:0/559363425 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630012400 con 0x7fe638145530 2023-10-30T12:17:26.555 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:26.555 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:26.555 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe634ff9640 10 monclient: dump: 2023-10-30T12:17:26.555 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:26.556 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:26.556 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:26.556 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:26.556 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:26.556 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:26.557 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:26.557 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:26.557 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:26.557 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:26.557 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.534+0000 7fe6367fc640 4 set_mon_vals no callback set 2023-10-30T12:17:26.557 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.535+0000 7fe63e733640 10 monclient: _renew_subs 2023-10-30T12:17:26.558 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.535+0000 7fe63e733640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:26.558 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.535+0000 7fe63e733640 1 -- 172.21.15.148:0/559363425 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6381d6c10 con 0x7fe638145530 2023-10-30T12:17:26.558 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.535+0000 7fe63e733640 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:26.558 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.535+0000 7fe63e733640 10 monclient: _renew_subs 2023-10-30T12:17:26.558 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.535+0000 7fe63e733640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:26.559 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.535+0000 7fe63e733640 1 -- 172.21.15.148:0/559363425 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6381d70e0 con 0x7fe638145530 2023-10-30T12:17:26.559 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.535+0000 7fe63e733640 1 librados: init done 2023-10-30T12:17:26.559 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.536+0000 7fe63e733640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:26.559 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.537+0000 7fe63e733640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:26.559 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.537+0000 7fe63e733640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:26.560 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.537+0000 7fe63e733640 1 -- 172.21.15.148:0/559363425 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63813f650 con 0x7fe638145530 2023-10-30T12:17:26.571 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.569+0000 7fe634ff9640 1 -- 172.21.15.148:0/559363425 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fe630007070 con 0x7fe638145530 2023-10-30T12:17:26.571 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.569+0000 7fe634ff9640 1 --2- 172.21.15.148:0/559363425 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe62406c730 0x7fe62406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:26.572 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.570+0000 7fe637fff640 1 --2- 172.21.15.148:0/559363425 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe62406c730 0x7fe62406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:26.572 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.570+0000 7fe637fff640 10 monclient: get_auth_request con 0x7fe62406c730 auth_method 0 2023-10-30T12:17:26.572 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.570+0000 7fe637fff640 1 --2- 172.21.15.148:0/559363425 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe62406c730 0x7fe62406ebf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe6280097a0 tx=0x7fe62803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:26.573 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.571+0000 7fe634ff9640 10 client.4215.objecter ms_handle_connect 0x7fe62406c730 2023-10-30T12:17:26.759 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.757+0000 7fe634ff9640 1 -- 172.21.15.148:0/559363425 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe630057b10 con 0x7fe638145530 2023-10-30T12:17:26.759 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.757+0000 7fe634ff9640 10 client.4215.objecter ms_dispatch 0x7fe638000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:17:26.759 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.757+0000 7fe634ff9640 3 client.4215.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:17:26.760 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.757+0000 7fe634ff9640 3 client.4215.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:17:26.760 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.757+0000 7fe634ff9640 20 client.4215.objecter dump_active .. 0 homeless 2023-10-30T12:17:26.985 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.983+0000 7fe634ff9640 1 -- 172.21.15.148:0/559363425 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe63005bbb0 con 0x7fe638145530 2023-10-30T12:17:26.985 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.983+0000 7fe634ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:26.985 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:26.983+0000 7fe634ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:27.088 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:17.631 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:27.191 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.189+0000 7fe63e733640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:27.191 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.189+0000 7fe63e733640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:27.191 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.189+0000 7fe63e733640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:17:27.191 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.189+0000 7fe63e733640 1 -- 172.21.15.148:0/559363425 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6381d6c10 con 0x7fe638145530 2023-10-30T12:17:27.206 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.205+0000 7fe634ff9640 1 -- 172.21.15.148:0/559363425 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe63005fd60 con 0x7fe638145530 2023-10-30T12:17:27.207 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.205+0000 7fe634ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:27.207 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.205+0000 7fe634ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:27.207 INFO:teuthology.misc.health.smithi148.stdout: 2023-10-30T12:17:27.208 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":10,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:17:09.070671+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:17:27.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 10 librados: watch_flush enter 2023-10-30T12:17:27.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 10 librados: watch_flush exit 2023-10-30T12:17:27.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 20 client.4215.objecter shutdown clearing up homeless session... 2023-10-30T12:17:27.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 10 client.4215.objecter successfully canceled tick 2023-10-30T12:17:27.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 -- 172.21.15.148:0/559363425 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe62406c730 msgr2=0x7fe62406ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:27.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 --2- 172.21.15.148:0/559363425 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe62406c730 0x7fe62406ebf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe6280097a0 tx=0x7fe62803d040 comp rx=0 tx=0).stop 2023-10-30T12:17:27.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 10 monclient: shutdown 2023-10-30T12:17:27.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:27.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 -- 172.21.15.148:0/559363425 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe638145530 msgr2=0x7fe638141670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:27.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 --2- 172.21.15.148:0/559363425 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe638145530 0x7fe638141670 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe63000caa0 tx=0x7fe63000cf60 comp rx=0 tx=0).stop 2023-10-30T12:17:27.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 -- 172.21.15.148:0/559363425 shutdown_connections 2023-10-30T12:17:27.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 --2- 172.21.15.148:0/559363425 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe62406c730 0x7fe62406ebf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:27.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 --2- 172.21.15.148:0/559363425 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe638145530 0x7fe638141670 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:27.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 --2- 172.21.15.148:0/559363425 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe63813f1e0 0x7fe6380ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:27.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 --2- 172.21.15.148:0/559363425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6380b4590 0x7fe63813fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:27.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 -- 172.21.15.148:0/559363425 >> 172.21.15.148:0/559363425 conn(0x7fe638060bf0 msgr2=0x7fe6380b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:27.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 -- 172.21.15.148:0/559363425 shutdown_connections 2023-10-30T12:17:27.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 -- 172.21.15.148:0/559363425 wait complete. 2023-10-30T12:17:27.214 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:27.208+0000 7fe63e733640 1 librados: shutdown 2023-10-30T12:17:27.223 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:17:27.340 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:17.895 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:27.491 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:18.054 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:27.675 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:18.228 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:27.936 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:18.481 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:27.949 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:18.539 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:27.986 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:18.567 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:28.214 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:18.788 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:32.745 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:23.287 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:32.750 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:23.293 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:32.955 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:23.511 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:32.959 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:23.515 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:33.346 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:23.899 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:33.350 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:23.903 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:33.458 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:24.020 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:33.458 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:24.004 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:33.461 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:24.024 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:33.464 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:24.010 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:33.795 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:24.384 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:33.798 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:24.388 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:33.847 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:24.427 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:33.851 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:24.431 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:34.029 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:17:34.093 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:24.667 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:34.097 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:24.671 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:34.132 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.131+0000 7f84c27b0640 10 monclient: get_monmap_and_config 2023-10-30T12:17:34.133 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.131+0000 7f84c27b0640 10 monclient: build_initial_monmap 2023-10-30T12:17:34.133 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.131+0000 7f84c27b0640 10 monclient: monmap: 2023-10-30T12:17:34.133 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:34.133 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:34.134 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:17:34.132302+0000 2023-10-30T12:17:34.134 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:17:34.132302+0000 2023-10-30T12:17:34.134 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:34.134 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:34.134 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:34.135 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:34.135 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:34.135 INFO:teuthology.misc.health.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:34.135 INFO:teuthology.misc.health.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:34.135 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:34.137 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.132+0000 7f84c27b0640 1 Processor -- start 2023-10-30T12:17:34.137 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.132+0000 7f84c27b0640 1 -- start start 2023-10-30T12:17:34.138 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.132+0000 7f84c27b0640 10 monclient: init 2023-10-30T12:17:34.138 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:34.138 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-10-30T12:17:34.138 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84bc0b4590 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:34.138 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84bc145700 addr v1:172.21.15.148:6789/0 2023-10-30T12:17:34.139 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc14d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:34.139 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 10 monclient(hunting): picked mon.noname-b con 0x7f84bc13f3b0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:34.139 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc0aa540 con 0x7f84bc145700 2023-10-30T12:17:34.139 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc0aa7b0 con 0x7f84bc0b4590 2023-10-30T12:17:34.139 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:34.140 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc067160 con 0x7f84bc13f3b0 2023-10-30T12:17:34.140 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:34.140 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.133+0000 7f84c27b0640 10 monclient(hunting): authenticate will time out at 1546.338672s 2023-10-30T12:17:34.148 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.147+0000 7f84bbfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f84bc0b4590 0x7f84bc144ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:53120/0 (socket says 172.21.15.148:53120) 2023-10-30T12:17:34.148 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.147+0000 7f84bbfff640 1 -- 172.21.15.148:0/3264525476 learned_addr learned my addr 172.21.15.148:0/3264525476 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:17:34.173 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.171+0000 7f84c0d4d640 1 --2- 172.21.15.148:0/3264525476 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc14d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:34.173 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.171+0000 7f84c0d4d640 10 monclient(hunting): get_auth_request con 0x7f84bc13f3b0 auth_method 0 2023-10-30T12:17:34.174 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.171+0000 7f84c0d4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:34.174 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.171+0000 7f84c0d4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:34.174 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.171+0000 7f84c0d4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:34.174 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.173+0000 7f84c0d4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:34.175 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.173+0000 7f84c0d4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:34.175 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.173+0000 7f84c0d4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:34.177 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.175+0000 7f84c0d4d640 10 monclient(hunting): handle_auth_done global_id 4184 payload 274 2023-10-30T12:17:34.177 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.175+0000 7f84c0d4d640 1 -- 172.21.15.148:0/3264525476 >> v1:172.21.15.150:6789/0 conn(0x7f84bc0b4590 legacy=0x7f84bc144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:34.178 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.175+0000 7f84c0d4d640 1 -- 172.21.15.148:0/3264525476 >> v1:172.21.15.148:6789/0 conn(0x7f84bc145700 legacy=0x7f84bc13eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:34.178 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.175+0000 7f84c0d4d640 10 monclient: _finish_hunting 0 2023-10-30T12:17:34.178 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.175+0000 7f84c0d4d640 1 monclient: found mon.noname-b 2023-10-30T12:17:34.178 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.175+0000 7f84c0d4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:34.179 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.175+0000 7f84c0d4d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:17:34.179 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.175+0000 7f84c0d4d640 1 -- 172.21.15.148:0/3264525476 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc0a88b0 con 0x7f84bc13f3b0 2023-10-30T12:17:34.181 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.180+0000 7f84c0d4d640 1 --2- 172.21.15.148:0/3264525476 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc14d370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009dc0 tx=0x7f84b402be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=322963da2e1021c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:34.187 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 1 -- 172.21.15.148:0/3264525476 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b402cbb0 con 0x7f84bc13f3b0 2023-10-30T12:17:34.187 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:34.187 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:34.188 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 10 monclient: dump: 2023-10-30T12:17:34.188 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:34.188 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:34.188 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:34.189 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:34.189 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:34.189 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:34.189 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:34.189 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:34.190 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:34.191 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:34.191 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 10 monclient: _finish_auth 0 2023-10-30T12:17:34.191 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:34.191 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:34.191 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 1 -- 172.21.15.148:0/3264525476 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b402cd20 con 0x7f84bc13f3b0 2023-10-30T12:17:34.192 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:34.192 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 1 -- 172.21.15.148:0/3264525476 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4034660 con 0x7f84bc13f3b0 2023-10-30T12:17:34.192 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:34.192 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:34.192 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.185+0000 7f84baffd640 10 monclient: dump: 2023-10-30T12:17:34.193 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:34.193 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:34.193 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:34.193 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:34.193 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:34.193 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:34.194 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:34.194 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:34.194 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:34.194 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:34.195 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.186+0000 7f84c27b0640 5 monclient: authenticate success, global_id 4184 2023-10-30T12:17:34.195 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.186+0000 7f84c27b0640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:34.195 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.186+0000 7f84c27b0640 4 set_mon_vals no callback set 2023-10-30T12:17:34.201 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.186+0000 7f84c27b0640 10 monclient: shutdown 2023-10-30T12:17:34.202 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.186+0000 7f84c27b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:34.202 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.186+0000 7f84c27b0640 1 -- 172.21.15.148:0/3264525476 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 msgr2=0x7f84bc14d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:34.202 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.186+0000 7f84c27b0640 1 --2- 172.21.15.148:0/3264525476 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc14d370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009dc0 tx=0x7f84b402be00 comp rx=0 tx=0).stop 2023-10-30T12:17:34.202 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.187+0000 7f84c27b0640 1 -- 172.21.15.148:0/3264525476 shutdown_connections 2023-10-30T12:17:34.202 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.187+0000 7f84c27b0640 1 --2- 172.21.15.148:0/3264525476 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc14d370 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:34.203 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.187+0000 7f84c27b0640 1 -- 172.21.15.148:0/3264525476 >> 172.21.15.148:0/3264525476 conn(0x7f84bc060bf0 msgr2=0x7f84bc061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:34.203 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.187+0000 7f84c27b0640 1 -- 172.21.15.148:0/3264525476 shutdown_connections 2023-10-30T12:17:34.203 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.187+0000 7f84c27b0640 1 -- 172.21.15.148:0/3264525476 wait complete. 2023-10-30T12:17:34.203 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.188+0000 7f84c27b0640 10 monclient: build_initial_monmap 2023-10-30T12:17:34.203 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.188+0000 7f84c27b0640 10 monclient: monmap: 2023-10-30T12:17:34.204 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:34.204 INFO:teuthology.misc.health.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:34.204 INFO:teuthology.misc.health.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:34.204 INFO:teuthology.misc.health.smithi148.stderr:created 0.000000 2023-10-30T12:17:34.204 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:34.204 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:34.205 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:34.205 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:34.205 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:34.205 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:34.205 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.188+0000 7f84c27b0640 1 librados: starting msgr at 2023-10-30T12:17:34.206 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.188+0000 7f84c27b0640 1 librados: starting objecter 2023-10-30T12:17:34.206 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.188+0000 7f84c27b0640 1 Processor -- start 2023-10-30T12:17:34.206 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.189+0000 7f84c27b0640 1 -- start start 2023-10-30T12:17:34.206 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.189+0000 7f84c27b0640 1 librados: setting wanted keys 2023-10-30T12:17:34.206 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.189+0000 7f84c27b0640 1 librados: calling monclient init 2023-10-30T12:17:34.207 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.189+0000 7f84c27b0640 10 monclient: init 2023-10-30T12:17:34.207 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:34.207 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:17:34.207 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84bc0b4590 0x7f84bc13aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:34.207 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 10 monclient(hunting): picked mon.noname-b con 0x7f84bc0b4590 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:34.207 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc0ab300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:34.208 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 10 monclient(hunting): picked mon.noname-c con 0x7f84bc13f3b0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:34.208 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84bc145700 0x7f84bc0ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:34.208 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 10 monclient(hunting): picked mon.noname-a con 0x7f84bc145700 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:34.208 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:34.208 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc145cd0 con 0x7f84bc145700 2023-10-30T12:17:34.209 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:34.209 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0ab840 con 0x7f84bc0b4590 2023-10-30T12:17:34.209 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.190+0000 7f84c27b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:34.209 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84c27b0640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0aaec0 con 0x7f84bc13f3b0 2023-10-30T12:17:34.209 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84bc0b4590 0x7f84bc13aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:34.209 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bb7fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc0ab300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:34.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84c27b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:34.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84c27b0640 10 monclient(hunting): authenticate will time out at 1546.396167s 2023-10-30T12:17:34.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84c0d4d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84bc145700 0x7f84bc0ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:34.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bb7fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc0ab300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:36170/0 (socket says 172.21.15.148:36170) 2023-10-30T12:17:34.210 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bb7fe640 1 -- 172.21.15.148:0/2672827431 learned_addr learned my addr 172.21.15.148:0/2672827431 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:34.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bb7fe640 10 monclient(hunting): get_auth_request con 0x7f84bc13f3b0 auth_method 0 2023-10-30T12:17:34.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:34.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:34.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:34.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84c0d4d640 10 monclient(hunting): get_auth_request con 0x7f84bc145700 auth_method 0 2023-10-30T12:17:34.211 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84c0d4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:34.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84c0d4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:34.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84c0d4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:34.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:34.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:34.212 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.191+0000 7f84bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:34.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.192+0000 7f84bbfff640 10 monclient(hunting): get_auth_request con 0x7f84bc0b4590 auth_method 0 2023-10-30T12:17:34.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.192+0000 7f84bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:34.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.192+0000 7f84bbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:34.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.192+0000 7f84bbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:34.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.192+0000 7f84c0d4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:34.213 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.192+0000 7f84c0d4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:34.214 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.192+0000 7f84c0d4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:34.214 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.192+0000 7f84bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:34.214 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.192+0000 7f84bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:34.214 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.192+0000 7f84bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:34.214 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.195+0000 7f84bb7fe640 10 monclient(hunting): handle_auth_done global_id 4187 payload 824 2023-10-30T12:17:34.215 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.195+0000 7f84bb7fe640 1 -- 172.21.15.148:0/2672827431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84bc0b4590 msgr2=0x7f84bc13aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:34.215 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.195+0000 7f84bb7fe640 1 --2- 172.21.15.148:0/2672827431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84bc0b4590 0x7f84bc13aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:34.215 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.195+0000 7f84bb7fe640 1 -- 172.21.15.148:0/2672827431 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84bc145700 msgr2=0x7f84bc0ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:34.215 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.195+0000 7f84bb7fe640 1 --2- 172.21.15.148:0/2672827431 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84bc145700 0x7f84bc0ad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:34.215 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.195+0000 7f84bb7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:17:34.215 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.195+0000 7f84bb7fe640 1 monclient: found mon.noname-c 2023-10-30T12:17:34.216 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.195+0000 7f84bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:34.216 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.195+0000 7f84bb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:17:34.216 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.195+0000 7f84bb7fe640 1 -- 172.21.15.148:0/2672827431 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc0acd20 con 0x7f84bc13f3b0 2023-10-30T12:17:34.216 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.198+0000 7f84bb7fe640 1 --2- 172.21.15.148:0/2672827431 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc0ab300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00b3e0 tx=0x7f84ac00b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:34.216 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.198+0000 7f84b8ff9640 10 client.?.objecter ms_handle_connect 0x7f84bc13f3b0 2023-10-30T12:17:34.217 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.198+0000 7f84b8ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:34.219 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.201+0000 7f84b8ff9640 1 -- 172.21.15.148:0/2672827431 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac011830 con 0x7f84bc13f3b0 2023-10-30T12:17:34.219 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.201+0000 7f84b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:34.220 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.201+0000 7f84b8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:34.220 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.201+0000 7f84b8ff9640 10 monclient: dump: 2023-10-30T12:17:34.220 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:34.220 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:34.220 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:34.221 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:34.221 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:34.221 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:34.221 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:34.221 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:34.221 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:34.222 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:34.222 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.201+0000 7f84b8ff9640 10 monclient: _finish_auth 0 2023-10-30T12:17:34.222 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.201+0000 7f84b8ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:17:34.222 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.201+0000 7f84b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:34.222 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.201+0000 7f84b8ff9640 1 -- 172.21.15.148:0/2672827431 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac00d040 con 0x7f84bc13f3b0 2023-10-30T12:17:34.223 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.201+0000 7f84b8ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:34.223 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.201+0000 7f84ba7fc640 4 set_mon_vals no callback set 2023-10-30T12:17:34.223 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.202+0000 7f84b8ff9640 1 -- 172.21.15.148:0/2672827431 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac011c00 con 0x7f84bc13f3b0 2023-10-30T12:17:34.223 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.202+0000 7f84c27b0640 5 monclient: authenticate success, global_id 4187 2023-10-30T12:17:34.223 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.202+0000 7f84c27b0640 10 monclient: _renew_subs 2023-10-30T12:17:34.223 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.202+0000 7f84c27b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:34.224 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.202+0000 7f84c27b0640 1 -- 172.21.15.148:0/2672827431 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84bc144c30 con 0x7f84bc13f3b0 2023-10-30T12:17:34.224 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.203+0000 7f84b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:34.224 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.203+0000 7f84b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:34.224 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.203+0000 7f84b8ff9640 10 monclient: dump: 2023-10-30T12:17:34.224 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:34.225 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:34.225 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:34.225 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:34.225 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:34.225 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:34.226 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:34.226 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:34.226 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:34.226 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:34.226 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.203+0000 7f84c27b0640 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:34.226 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.203+0000 7f84c27b0640 10 monclient: _renew_subs 2023-10-30T12:17:34.227 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.203+0000 7f84c27b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:34.227 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.203+0000 7f84c27b0640 1 -- 172.21.15.148:0/2672827431 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc0ad7e0 con 0x7f84bc13f3b0 2023-10-30T12:17:34.227 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.203+0000 7f84c27b0640 1 librados: init done 2023-10-30T12:17:34.227 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.204+0000 7f84c27b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:34.227 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.205+0000 7f84c27b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:34.228 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.205+0000 7f84c27b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:34.228 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.205+0000 7f84c27b0640 1 -- 172.21.15.148:0/2672827431 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84bc0ab020 con 0x7f84bc13f3b0 2023-10-30T12:17:34.466 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.465+0000 7f84b8ff9640 1 -- 172.21.15.148:0/2672827431 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7f84ac01a400 con 0x7f84bc13f3b0 2023-10-30T12:17:34.467 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.465+0000 7f84b8ff9640 1 --2- 172.21.15.148:0/2672827431 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84a806c730 0x7f84a806ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:34.468 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.465+0000 7f84bbfff640 1 --2- 172.21.15.148:0/2672827431 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84a806c730 0x7f84a806ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:34.468 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.466+0000 7f84bbfff640 10 monclient: get_auth_request con 0x7f84a806c730 auth_method 0 2023-10-30T12:17:34.468 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.466+0000 7f84bbfff640 1 --2- 172.21.15.148:0/2672827431 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84a806c730 0x7f84a806ebf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f84b0005880 tx=0x7f84b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:34.468 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.466+0000 7f84b8ff9640 10 client.4187.objecter ms_handle_connect 0x7f84a806c730 2023-10-30T12:17:34.494 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.492+0000 7f84b8ff9640 1 -- 172.21.15.148:0/2672827431 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f84ac0584b0 con 0x7f84bc13f3b0 2023-10-30T12:17:34.495 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.493+0000 7f84b8ff9640 10 client.4187.objecter ms_dispatch 0x7f84bc000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:17:34.495 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.493+0000 7f84b8ff9640 3 client.4187.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:17:34.495 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.493+0000 7f84b8ff9640 3 client.4187.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:17:34.495 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.493+0000 7f84b8ff9640 20 client.4187.objecter dump_active .. 0 homeless 2023-10-30T12:17:34.566 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.565+0000 7f84b8ff9640 1 -- 172.21.15.148:0/2672827431 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f84bc144c30 con 0x7f84bc13f3b0 2023-10-30T12:17:34.567 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.565+0000 7f84b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:34.567 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.565+0000 7f84b8ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:34.795 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.793+0000 7f84c27b0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:34.795 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.793+0000 7f84c27b0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:34.795 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.793+0000 7f84c27b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:34.795 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.793+0000 7f84c27b0640 1 -- 172.21.15.148:0/2672827431 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84bc0ad7e0 con 0x7f84bc13f3b0 2023-10-30T12:17:34.896 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.895+0000 7f84b8ff9640 1 -- 172.21.15.148:0/2672827431 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f84ac05c550 con 0x7f84bc13f3b0 2023-10-30T12:17:34.897 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.895+0000 7f84b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:34.897 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.895+0000 7f84b8ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:34.897 INFO:teuthology.misc.health.smithi148.stdout: 2023-10-30T12:17:34.897 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":10,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:17:09.070671+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:17:34.899 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 10 librados: watch_flush enter 2023-10-30T12:17:34.899 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 10 librados: watch_flush exit 2023-10-30T12:17:34.899 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 20 client.4187.objecter shutdown clearing up homeless session... 2023-10-30T12:17:34.899 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 10 client.4187.objecter successfully canceled tick 2023-10-30T12:17:34.900 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 1 -- 172.21.15.148:0/2672827431 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84a806c730 msgr2=0x7f84a806ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:34.900 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 1 --2- 172.21.15.148:0/2672827431 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84a806c730 0x7f84a806ebf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f84b0005880 tx=0x7f84b0009040 comp rx=0 tx=0).stop 2023-10-30T12:17:34.900 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 10 monclient: shutdown 2023-10-30T12:17:34.900 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:34.901 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 1 -- 172.21.15.148:0/2672827431 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 msgr2=0x7f84bc0ab300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:34.901 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 1 --2- 172.21.15.148:0/2672827431 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc0ab300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00b3e0 tx=0x7f84ac00b8a0 comp rx=0 tx=0).stop 2023-10-30T12:17:34.901 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 1 -- 172.21.15.148:0/2672827431 shutdown_connections 2023-10-30T12:17:34.901 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 1 --2- 172.21.15.148:0/2672827431 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84a806c730 0x7f84a806ebf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:34.901 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 1 --2- 172.21.15.148:0/2672827431 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84bc145700 0x7f84bc0ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:34.902 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 1 --2- 172.21.15.148:0/2672827431 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f84bc13f3b0 0x7f84bc0ab300 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:34.902 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 1 --2- 172.21.15.148:0/2672827431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84bc0b4590 0x7f84bc13aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:34.902 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.897+0000 7f84c27b0640 1 -- 172.21.15.148:0/2672827431 >> 172.21.15.148:0/2672827431 conn(0x7f84bc060bf0 msgr2=0x7f84bc0738f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:34.903 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.898+0000 7f84c27b0640 1 -- 172.21.15.148:0/2672827431 shutdown_connections 2023-10-30T12:17:34.903 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.899+0000 7f84c27b0640 1 -- 172.21.15.148:0/2672827431 wait complete. 2023-10-30T12:17:34.903 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:34.899+0000 7f84c27b0640 1 librados: shutdown 2023-10-30T12:17:34.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:17:36.871 INFO:tasks.ceph.osd.6.smithi150.stderr:==00:00:00:27.413 86375== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:37.774 INFO:tasks.ceph.osd.5.smithi150.stderr:==00:00:00:28.327 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:37.886 INFO:tasks.ceph.osd.4.smithi150.stderr:==00:00:00:28.441 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:37.892 INFO:tasks.ceph.osd.7.smithi150.stderr:==00:00:00:28.437 86374== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:37.938 INFO:tasks.ceph.osd.2.smithi148.stderr:==00:00:00:28.501 86516== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:38.303 INFO:tasks.ceph.osd.0.smithi148.stderr:==00:00:00:28.892 86502== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:38.407 INFO:tasks.ceph.osd.1.smithi148.stderr:==00:00:00:28.985 86511== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:38.635 INFO:tasks.ceph.osd.3.smithi148.stderr:==00:00:00:29.208 86512== Warning: unimplemented fcntl command: 1038 2023-10-30T12:17:41.718 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:17:41.826 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.824+0000 7f23d0361640 10 monclient: get_monmap_and_config 2023-10-30T12:17:41.826 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.824+0000 7f23d0361640 10 monclient: build_initial_monmap 2023-10-30T12:17:41.826 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.824+0000 7f23d0361640 10 monclient: monmap: 2023-10-30T12:17:41.827 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:41.827 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:41.827 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:17:41.825845+0000 2023-10-30T12:17:41.827 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:17:41.825845+0000 2023-10-30T12:17:41.827 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:41.828 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:41.828 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:41.828 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:41.828 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:41.828 INFO:teuthology.misc.health.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:41.828 INFO:teuthology.misc.health.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:41.829 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:41.831 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.825+0000 7f23d0361640 1 Processor -- start 2023-10-30T12:17:41.831 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.825+0000 7f23d0361640 1 -- start start 2023-10-30T12:17:41.831 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.825+0000 7f23d0361640 10 monclient: init 2023-10-30T12:17:41.831 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:41.832 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-10-30T12:17:41.832 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23c80b4590 addr v1:172.21.15.148:6789/0 2023-10-30T12:17:41.832 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f23c8145700 0x7f23c813eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:41.832 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 10 monclient(hunting): picked mon.noname-a con 0x7f23c8145700 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:41.832 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f23c813f1e0 0x7f23c814c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:41.833 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 10 monclient(hunting): picked mon.noname-c con 0x7f23c813f1e0 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:41.833 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80aa540 con 0x7f23c80b4590 2023-10-30T12:17:41.833 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:41.833 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c8067160 con 0x7f23c8145700 2023-10-30T12:17:41.833 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:41.834 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c80aa7b0 con 0x7f23c813f1e0 2023-10-30T12:17:41.834 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:41.834 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23d0361640 10 monclient(hunting): authenticate will time out at 1554.031433s 2023-10-30T12:17:41.834 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23ce0fd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f23c80b4590 0x7f23c8144ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:33224/0 (socket says 172.21.15.148:33224) 2023-10-30T12:17:41.834 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.826+0000 7f23ce0fd640 1 -- 172.21.15.148:0/1734839540 learned_addr learned my addr 172.21.15.148:0/1734839540 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:17:41.835 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.827+0000 7f23ce8fe640 1 --2- 172.21.15.148:0/1734839540 >> v2:172.21.15.150:3300/0 conn(0x7f23c813f1e0 0x7f23c814c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:41.835 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.827+0000 7f23ce8fe640 10 monclient(hunting): get_auth_request con 0x7f23c813f1e0 auth_method 0 2023-10-30T12:17:41.835 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.827+0000 7f23ce8fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:41.835 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.827+0000 7f23ce8fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:41.835 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.827+0000 7f23ce8fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:41.835 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.828+0000 7f23ce8fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:41.836 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.828+0000 7f23ce8fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:41.836 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.828+0000 7f23ce8fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:41.837 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.830+0000 7f23cd8fc640 1 --2- 172.21.15.148:0/1734839540 >> v2:172.21.15.148:3300/0 conn(0x7f23c8145700 0x7f23c813eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:41.837 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.830+0000 7f23ce8fe640 10 monclient(hunting): handle_auth_done global_id 4222 payload 274 2023-10-30T12:17:41.837 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.830+0000 7f23ce8fe640 1 -- 172.21.15.148:0/1734839540 >> v2:172.21.15.148:3300/0 conn(0x7f23c8145700 msgr2=0x7f23c813eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:41.837 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.830+0000 7f23ce8fe640 1 --2- 172.21.15.148:0/1734839540 >> v2:172.21.15.148:3300/0 conn(0x7f23c8145700 0x7f23c813eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:41.837 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.830+0000 7f23ce8fe640 1 -- 172.21.15.148:0/1734839540 >> v1:172.21.15.148:6789/0 conn(0x7f23c80b4590 legacy=0x7f23c8144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:41.838 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.830+0000 7f23ce8fe640 10 monclient: _finish_hunting 0 2023-10-30T12:17:41.838 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.830+0000 7f23ce8fe640 1 monclient: found mon.noname-c 2023-10-30T12:17:41.838 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.830+0000 7f23ce8fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:41.838 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.830+0000 7f23ce8fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:17:41.838 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.830+0000 7f23ce8fe640 1 -- 172.21.15.148:0/1734839540 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80a88b0 con 0x7f23c813f1e0 2023-10-30T12:17:41.839 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.833+0000 7f23ce8fe640 1 --2- 172.21.15.148:0/1734839540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c813f1e0 0x7f23c814c360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f23c80b2f50 tx=0x7f23c002be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acde2006229b123e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:41.841 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.838+0000 7f23cd0fb640 1 -- 172.21.15.148:0/1734839540 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c002cbe0 con 0x7f23c813f1e0 2023-10-30T12:17:41.841 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.838+0000 7f23cd0fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:41.842 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.838+0000 7f23cd0fb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:41.842 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.838+0000 7f23cd0fb640 10 monclient: dump: 2023-10-30T12:17:41.842 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:41.842 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:41.843 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:41.843 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:41.843 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:41.843 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:41.843 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:41.844 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:41.844 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:41.844 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:41.844 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23cd0fb640 10 monclient: _finish_auth 0 2023-10-30T12:17:41.844 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23cd0fb640 10 monclient: _check_auth_tickets 2023-10-30T12:17:41.844 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23cd0fb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:41.845 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 5 monclient: authenticate success, global_id 4222 2023-10-30T12:17:41.845 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:41.845 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23cd0fb640 1 -- 172.21.15.148:0/1734839540 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c002cd50 con 0x7f23c813f1e0 2023-10-30T12:17:41.845 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23cd0fb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:41.845 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:41.846 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 4 set_mon_vals no callback set 2023-10-30T12:17:41.846 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23cd0fb640 1 -- 172.21.15.148:0/1734839540 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c0034660 con 0x7f23c813f1e0 2023-10-30T12:17:41.846 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23cd0fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:41.846 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23cd0fb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:41.846 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23cd0fb640 10 monclient: dump: 2023-10-30T12:17:41.847 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:41.847 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:41.847 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:41.847 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:41.847 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:41.848 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:41.848 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:41.848 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:41.848 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:41.848 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:41.849 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 10 monclient: shutdown 2023-10-30T12:17:41.849 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:41.849 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 1 -- 172.21.15.148:0/1734839540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c813f1e0 msgr2=0x7f23c814c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:41.849 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 1 --2- 172.21.15.148:0/1734839540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c813f1e0 0x7f23c814c360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f23c80b2f50 tx=0x7f23c002be30 comp rx=0 tx=0).stop 2023-10-30T12:17:41.849 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 1 -- 172.21.15.148:0/1734839540 shutdown_connections 2023-10-30T12:17:41.849 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 1 --2- 172.21.15.148:0/1734839540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c813f1e0 0x7f23c814c360 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:41.850 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 1 --2- 172.21.15.148:0/1734839540 >> v2:172.21.15.148:3300/0 conn(0x7f23c8145700 0x7f23c813eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:41.850 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.839+0000 7f23d0361640 1 -- 172.21.15.148:0/1734839540 >> 172.21.15.148:0/1734839540 conn(0x7f23c8060bf0 msgr2=0x7f23c8061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:41.850 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.840+0000 7f23d0361640 1 -- 172.21.15.148:0/1734839540 shutdown_connections 2023-10-30T12:17:41.850 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.840+0000 7f23d0361640 1 -- 172.21.15.148:0/1734839540 wait complete. 2023-10-30T12:17:41.856 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.841+0000 7f23d0361640 10 monclient: build_initial_monmap 2023-10-30T12:17:41.856 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.841+0000 7f23d0361640 10 monclient: monmap: 2023-10-30T12:17:41.856 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:41.856 INFO:teuthology.misc.health.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:41.857 INFO:teuthology.misc.health.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:41.857 INFO:teuthology.misc.health.smithi148.stderr:created 0.000000 2023-10-30T12:17:41.857 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:41.857 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:41.858 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:41.858 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:41.858 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:41.858 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:41.858 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.841+0000 7f23d0361640 1 librados: starting msgr at 2023-10-30T12:17:41.859 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.841+0000 7f23d0361640 1 librados: starting objecter 2023-10-30T12:17:41.859 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.841+0000 7f23d0361640 1 Processor -- start 2023-10-30T12:17:41.859 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.841+0000 7f23d0361640 1 -- start start 2023-10-30T12:17:41.859 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.841+0000 7f23d0361640 1 librados: setting wanted keys 2023-10-30T12:17:41.859 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.842+0000 7f23d0361640 1 librados: calling monclient init 2023-10-30T12:17:41.860 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.842+0000 7f23d0361640 10 monclient: init 2023-10-30T12:17:41.860 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.842+0000 7f23d0361640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:41.860 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:17:41.860 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c80b4590 0x7f23c813afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:41.860 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 10 monclient(hunting): picked mon.noname-b con 0x7f23c80b4590 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:41.860 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c813f1e0 0x7f23c80ad320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:41.861 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 10 monclient(hunting): picked mon.noname-a con 0x7f23c813f1e0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:41.861 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23c8145700 0x7f23c80ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:41.861 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 10 monclient(hunting): picked mon.noname-c con 0x7f23c8145700 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:41.861 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:41.861 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80a8d40 con 0x7f23c813f1e0 2023-10-30T12:17:41.862 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:41.862 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80acf70 con 0x7f23c80b4590 2023-10-30T12:17:41.862 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:41.862 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80ab930 con 0x7f23c8145700 2023-10-30T12:17:41.862 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:41.862 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23ce0fd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c80b4590 0x7f23c813afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:41.863 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23d0361640 10 monclient(hunting): authenticate will time out at 1554.048800s 2023-10-30T12:17:41.863 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.843+0000 7f23cd8fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c813f1e0 0x7f23c80ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:41.863 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23ce8fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23c8145700 0x7f23c80ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:41.863 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23cd8fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c813f1e0 0x7f23c80ad320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35106/0 (socket says 172.21.15.148:35106) 2023-10-30T12:17:41.863 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23cd8fc640 1 -- 172.21.15.148:0/620936264 learned_addr learned my addr 172.21.15.148:0/620936264 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:41.864 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23cd8fc640 10 monclient(hunting): get_auth_request con 0x7f23c813f1e0 auth_method 0 2023-10-30T12:17:41.864 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23cd8fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:41.864 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23cd8fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:41.864 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23cd8fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:41.864 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23ce0fd640 10 monclient(hunting): get_auth_request con 0x7f23c80b4590 auth_method 0 2023-10-30T12:17:41.865 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23ce0fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:41.865 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23ce0fd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:41.865 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23ce0fd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:41.865 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23ce8fe640 10 monclient(hunting): get_auth_request con 0x7f23c8145700 auth_method 0 2023-10-30T12:17:41.865 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23ce8fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:41.865 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23ce8fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:41.866 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23ce8fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:41.866 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23cd8fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:41.866 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.844+0000 7f23cd8fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:41.866 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.845+0000 7f23cd8fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:41.866 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.845+0000 7f23ce8fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:41.867 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.845+0000 7f23ce8fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:41.867 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.845+0000 7f23ce8fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:41.867 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.845+0000 7f23ce0fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:41.867 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.845+0000 7f23ce0fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:41.867 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.845+0000 7f23ce0fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:41.867 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.849+0000 7f23ce0fd640 10 monclient(hunting): handle_auth_done global_id 4225 payload 824 2023-10-30T12:17:41.868 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.849+0000 7f23ce0fd640 1 -- 172.21.15.148:0/620936264 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23c8145700 msgr2=0x7f23c80ab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:41.868 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.849+0000 7f23ce0fd640 1 --2- 172.21.15.148:0/620936264 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23c8145700 0x7f23c80ab3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:41.868 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.849+0000 7f23ce0fd640 1 -- 172.21.15.148:0/620936264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c813f1e0 msgr2=0x7f23c80ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:41.868 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.849+0000 7f23ce0fd640 1 --2- 172.21.15.148:0/620936264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c813f1e0 0x7f23c80ad320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:41.868 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.849+0000 7f23ce0fd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:41.869 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.849+0000 7f23ce0fd640 1 monclient: found mon.noname-b 2023-10-30T12:17:41.869 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.849+0000 7f23ce0fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:41.869 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.849+0000 7f23ce0fd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:41.869 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.849+0000 7f23ce0fd640 1 -- 172.21.15.148:0/620936264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80ab160 con 0x7f23c80b4590 2023-10-30T12:17:41.869 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.852+0000 7f23ce0fd640 1 --2- 172.21.15.148:0/620936264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c80b4590 0x7f23c813afa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f23c400c470 tx=0x7f23c400c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:41.870 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.852+0000 7f23beffd640 10 client.?.objecter ms_handle_connect 0x7f23c80b4590 2023-10-30T12:17:41.870 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.852+0000 7f23beffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:41.871 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23beffd640 1 -- 172.21.15.148:0/620936264 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4005ce0 con 0x7f23c80b4590 2023-10-30T12:17:41.871 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:41.872 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:41.872 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23beffd640 10 monclient: dump: 2023-10-30T12:17:41.872 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:41.872 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:41.872 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:41.873 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:41.873 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:41.873 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:41.873 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:41.873 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:41.873 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:41.874 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:41.874 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23beffd640 10 monclient: _finish_auth 0 2023-10-30T12:17:41.874 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23beffd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:41.874 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:41.874 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23d0361640 5 monclient: authenticate success, global_id 4225 2023-10-30T12:17:41.875 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23d0361640 10 monclient: _renew_subs 2023-10-30T12:17:41.875 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23d0361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:41.875 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23d0361640 1 -- 172.21.15.148:0/620936264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c81442d0 con 0x7f23c80b4590 2023-10-30T12:17:41.875 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23d0361640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:41.875 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23d0361640 10 monclient: _renew_subs 2023-10-30T12:17:41.876 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23d0361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:41.876 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23d0361640 1 -- 172.21.15.148:0/620936264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c80abb70 con 0x7f23c80b4590 2023-10-30T12:17:41.876 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.855+0000 7f23d0361640 1 librados: init done 2023-10-30T12:17:41.876 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.856+0000 7f23d0361640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:41.876 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.856+0000 7f23d0361640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:41.876 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.856+0000 7f23d0361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:41.877 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.856+0000 7f23d0361640 1 -- 172.21.15.148:0/620936264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c81450d0 con 0x7f23c80b4590 2023-10-30T12:17:41.877 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.857+0000 7f23beffd640 1 -- 172.21.15.148:0/620936264 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c400e040 con 0x7f23c80b4590 2023-10-30T12:17:41.877 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.857+0000 7f23beffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:41.877 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.857+0000 7f23cc8fa640 4 set_mon_vals no callback set 2023-10-30T12:17:41.877 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.858+0000 7f23beffd640 1 -- 172.21.15.148:0/620936264 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4007270 con 0x7f23c80b4590 2023-10-30T12:17:41.878 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.858+0000 7f23beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:41.878 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.858+0000 7f23beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:41.878 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:41.858+0000 7f23beffd640 10 monclient: dump: 2023-10-30T12:17:41.878 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:41.878 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:41.878 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:41.879 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:41.879 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:41.879 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:41.879 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:41.879 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:41.880 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:41.880 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:42.251 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.249+0000 7f23beffd640 1 -- 172.21.15.148:0/620936264 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7f23c40073e0 con 0x7f23c80b4590 2023-10-30T12:17:42.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.250+0000 7f23beffd640 1 --2- 172.21.15.148:0/620936264 >> v2:172.21.15.148:6800/3205090179 conn(0x7f23ac06c730 0x7f23ac06ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:42.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.250+0000 7f23cd8fc640 1 --2- 172.21.15.148:0/620936264 >> v2:172.21.15.148:6800/3205090179 conn(0x7f23ac06c730 0x7f23ac06ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:42.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.250+0000 7f23cd8fc640 10 monclient: get_auth_request con 0x7f23ac06c730 auth_method 0 2023-10-30T12:17:42.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.250+0000 7f23beffd640 10 client.4225.objecter ms_handle_connect 0x7f23ac06c730 2023-10-30T12:17:42.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.250+0000 7f23cd8fc640 1 --2- 172.21.15.148:0/620936264 >> v2:172.21.15.148:6800/3205090179 conn(0x7f23ac06c730 0x7f23ac06ebf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f23c8073da0 tx=0x7f23b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:42.256 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.255+0000 7f23beffd640 1 -- 172.21.15.148:0/620936264 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f23c40576e0 con 0x7f23c80b4590 2023-10-30T12:17:42.256 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.255+0000 7f23beffd640 10 client.4225.objecter ms_dispatch 0x7f23c8000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:17:42.256 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.255+0000 7f23beffd640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:17:42.257 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.255+0000 7f23beffd640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:17:42.257 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.255+0000 7f23beffd640 20 client.4225.objecter dump_active .. 0 homeless 2023-10-30T12:17:42.274 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.273+0000 7f23beffd640 1 -- 172.21.15.148:0/620936264 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f23c405b780 con 0x7f23c80b4590 2023-10-30T12:17:42.274 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.273+0000 7f23beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:42.274 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.273+0000 7f23beffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:42.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.505+0000 7f23d0361640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:42.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.505+0000 7f23d0361640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:42.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.505+0000 7f23d0361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:42.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.505+0000 7f23d0361640 1 -- 172.21.15.148:0/620936264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23c81442d0 con 0x7f23c80b4590 2023-10-30T12:17:42.535 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.534+0000 7f23beffd640 1 -- 172.21.15.148:0/620936264 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f23c405f930 con 0x7f23c80b4590 2023-10-30T12:17:42.536 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.534+0000 7f23beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:42.536 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.534+0000 7f23beffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:42.536 INFO:teuthology.misc.health.smithi148.stdout: 2023-10-30T12:17:42.536 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":10,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:17:09.070671+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:17:42.537 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 10 librados: watch_flush enter 2023-10-30T12:17:42.538 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 10 librados: watch_flush exit 2023-10-30T12:17:42.539 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 20 client.4225.objecter shutdown clearing up homeless session... 2023-10-30T12:17:42.539 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 10 client.4225.objecter successfully canceled tick 2023-10-30T12:17:42.539 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 -- 172.21.15.148:0/620936264 >> v2:172.21.15.148:6800/3205090179 conn(0x7f23ac06c730 msgr2=0x7f23ac06ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:42.539 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 --2- 172.21.15.148:0/620936264 >> v2:172.21.15.148:6800/3205090179 conn(0x7f23ac06c730 0x7f23ac06ebf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f23c8073da0 tx=0x7f23b8008040 comp rx=0 tx=0).stop 2023-10-30T12:17:42.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 10 monclient: shutdown 2023-10-30T12:17:42.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:42.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 -- 172.21.15.148:0/620936264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c80b4590 msgr2=0x7f23c813afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:42.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 --2- 172.21.15.148:0/620936264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c80b4590 0x7f23c813afa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f23c400c470 tx=0x7f23c400c930 comp rx=0 tx=0).stop 2023-10-30T12:17:42.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 -- 172.21.15.148:0/620936264 shutdown_connections 2023-10-30T12:17:42.540 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 --2- 172.21.15.148:0/620936264 >> v2:172.21.15.148:6800/3205090179 conn(0x7f23ac06c730 0x7f23ac06ebf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:42.541 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 --2- 172.21.15.148:0/620936264 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23c8145700 0x7f23c80ab3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:42.541 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 --2- 172.21.15.148:0/620936264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c813f1e0 0x7f23c80ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:42.541 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 --2- 172.21.15.148:0/620936264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c80b4590 0x7f23c813afa0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:42.541 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 -- 172.21.15.148:0/620936264 >> 172.21.15.148:0/620936264 conn(0x7f23c8060bf0 msgr2=0x7f23c80b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:42.541 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.536+0000 7f23d0361640 1 -- 172.21.15.148:0/620936264 shutdown_connections 2023-10-30T12:17:42.542 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.537+0000 7f23d0361640 1 -- 172.21.15.148:0/620936264 wait complete. 2023-10-30T12:17:42.542 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:42.537+0000 7f23d0361640 1 librados: shutdown 2023-10-30T12:17:42.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:17:45.639 INFO:tasks.ceph.osd.6.smithi150.stderr:2023-10-30T12:17:45.635+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-10-30T12:17:46.195 INFO:tasks.ceph.osd.4.smithi150.stderr:2023-10-30T12:17:46.191+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-10-30T12:17:46.269 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:17:46.265+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-10-30T12:17:46.374 INFO:tasks.ceph.osd.7.smithi150.stderr:2023-10-30T12:17:46.369+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-10-30T12:17:46.675 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T12:17:46.669+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-10-30T12:17:46.693 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T12:17:46.688+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-10-30T12:17:46.919 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:17:46.869+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-10-30T12:17:46.940 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T12:17:46.935+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-10-30T12:17:48.596 INFO:tasks.ceph.osd.6.smithi150.stderr:2023-10-30T12:17:48.579+0000 ac7d640 -1 osd.6 0 waiting for initial osdmap 2023-10-30T12:17:48.988 INFO:tasks.ceph.osd.6.smithi150.stderr:2023-10-30T12:17:48.974+0000 12a5f640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:17:49.356 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:17:49.445 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.444+0000 7fce1371d640 10 monclient: get_monmap_and_config 2023-10-30T12:17:49.446 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.444+0000 7fce1371d640 10 monclient: build_initial_monmap 2023-10-30T12:17:49.446 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.444+0000 7fce1371d640 10 monclient: monmap: 2023-10-30T12:17:49.447 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:49.447 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:49.448 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:17:49.445149+0000 2023-10-30T12:17:49.448 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:17:49.445149+0000 2023-10-30T12:17:49.448 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:49.449 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:49.449 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:49.450 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:49.450 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:49.450 INFO:teuthology.misc.health.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:49.451 INFO:teuthology.misc.health.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:49.451 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:49.463 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.444+0000 7fce1371d640 1 Processor -- start 2023-10-30T12:17:49.464 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.444+0000 7fce1371d640 1 -- start start 2023-10-30T12:17:49.464 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.444+0000 7fce1371d640 10 monclient: init 2023-10-30T12:17:49.465 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:49.465 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-10-30T12:17:49.465 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c0b4590 0x7fce0c144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:49.466 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 10 monclient(hunting): picked mon.noname-b con 0x7fce0c0b4590 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:49.466 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce0c145530 0x7fce0c13eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:49.467 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 10 monclient(hunting): picked mon.noname-c con 0x7fce0c145530 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:49.467 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fce0c13f1e0 0x7fce0c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:49.467 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 10 monclient(hunting): picked mon.noname-a con 0x7fce0c13f1e0 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:49.468 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:49.468 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0c067160 con 0x7fce0c13f1e0 2023-10-30T12:17:49.468 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:49.469 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0c0aa540 con 0x7fce0c145530 2023-10-30T12:17:49.469 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:49.470 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c145990 con 0x7fce0c0b4590 2023-10-30T12:17:49.470 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:49.470 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.445+0000 7fce1371d640 10 monclient(hunting): authenticate will time out at 1561.650460s 2023-10-30T12:17:49.471 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce114b9640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c0b4590 0x7fce0c144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:49.471 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce114b9640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c0b4590 0x7fce0c144ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:36746/0 (socket says 172.21.15.148:36746) 2023-10-30T12:17:49.471 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce114b9640 1 -- 172.21.15.148:0/1498382278 learned_addr learned my addr 172.21.15.148:0/1498382278 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:49.472 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce114b9640 10 monclient(hunting): get_auth_request con 0x7fce0c0b4590 auth_method 0 2023-10-30T12:17:49.472 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce114b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:49.473 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce114b9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:49.473 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce114b9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:49.473 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce10cb8640 1 --2- 172.21.15.148:0/1498382278 >> v2:172.21.15.150:3300/0 conn(0x7fce0c145530 0x7fce0c13eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:49.474 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce10cb8640 10 monclient(hunting): get_auth_request con 0x7fce0c145530 auth_method 0 2023-10-30T12:17:49.474 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce10cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:49.475 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce10cb8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:49.475 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce10cb8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:49.476 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce114b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:49.476 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce114b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:49.476 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.446+0000 7fce114b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:49.477 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.447+0000 7fce10cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:49.477 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.447+0000 7fce10cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:49.478 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.447+0000 7fce10cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:49.478 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.448+0000 7fce114b9640 10 monclient(hunting): handle_auth_done global_id 4217 payload 274 2023-10-30T12:17:49.478 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.448+0000 7fce114b9640 1 -- 172.21.15.148:0/1498382278 >> v2:172.21.15.150:3300/0 conn(0x7fce0c145530 msgr2=0x7fce0c13eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:49.479 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.448+0000 7fce114b9640 1 --2- 172.21.15.148:0/1498382278 >> v2:172.21.15.150:3300/0 conn(0x7fce0c145530 0x7fce0c13eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:49.479 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.448+0000 7fce114b9640 1 -- 172.21.15.148:0/1498382278 >> v2:172.21.15.148:3300/0 conn(0x7fce0c13f1e0 msgr2=0x7fce0c0b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:49.479 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.448+0000 7fce114b9640 1 --2- 172.21.15.148:0/1498382278 >> v2:172.21.15.148:3300/0 conn(0x7fce0c13f1e0 0x7fce0c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:49.480 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.448+0000 7fce114b9640 10 monclient: _finish_hunting 0 2023-10-30T12:17:49.480 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.448+0000 7fce114b9640 1 monclient: found mon.noname-b 2023-10-30T12:17:49.481 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.448+0000 7fce114b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:49.481 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.448+0000 7fce114b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:17:49.481 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.448+0000 7fce114b9640 1 -- 172.21.15.148:0/1498382278 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c0aa7b0 con 0x7fce0c0b4590 2023-10-30T12:17:49.482 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.450+0000 7fce114b9640 1 --2- 172.21.15.148:0/1498382278 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c0b4590 0x7fce0c144ff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fce0c0b8440 tx=0x7fcdfc02b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=476165d17942f2e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:49.482 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 1 -- 172.21.15.148:0/1498382278 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0317e0 con 0x7fce0c0b4590 2023-10-30T12:17:49.482 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:49.483 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:49.483 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 10 monclient: dump: 2023-10-30T12:17:49.484 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:49.484 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:49.484 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:49.485 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:49.485 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:49.485 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:49.486 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:49.486 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:49.487 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:49.487 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:49.487 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 10 monclient: _finish_auth 0 2023-10-30T12:17:49.488 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 10 monclient: _check_auth_tickets 2023-10-30T12:17:49.488 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:49.488 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 1 -- 172.21.15.148:0/1498382278 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc031950 con 0x7fce0c0b4590 2023-10-30T12:17:49.489 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:49.489 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 1 -- 172.21.15.148:0/1498382278 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc031cf0 con 0x7fce0c0b4590 2023-10-30T12:17:49.490 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce1371d640 5 monclient: authenticate success, global_id 4217 2023-10-30T12:17:49.490 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce1371d640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:49.490 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce1371d640 4 set_mon_vals no callback set 2023-10-30T12:17:49.491 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:49.491 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:49.492 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.454+0000 7fce03fff640 10 monclient: dump: 2023-10-30T12:17:49.492 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:49.492 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:49.493 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:49.493 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:49.494 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:49.494 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:49.494 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:49.495 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:49.495 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:49.495 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:49.496 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 10 monclient: shutdown 2023-10-30T12:17:49.496 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:49.497 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 1 -- 172.21.15.148:0/1498382278 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c0b4590 msgr2=0x7fce0c144ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:49.497 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 1 --2- 172.21.15.148:0/1498382278 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c0b4590 0x7fce0c144ff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fce0c0b8440 tx=0x7fcdfc02b880 comp rx=0 tx=0).stop 2023-10-30T12:17:49.497 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 1 -- 172.21.15.148:0/1498382278 shutdown_connections 2023-10-30T12:17:49.498 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 1 --2- 172.21.15.148:0/1498382278 >> v2:172.21.15.148:3300/0 conn(0x7fce0c13f1e0 0x7fce0c0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:49.498 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 1 --2- 172.21.15.148:0/1498382278 >> v2:172.21.15.150:3300/0 conn(0x7fce0c145530 0x7fce0c13eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:49.499 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 1 --2- 172.21.15.148:0/1498382278 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c0b4590 0x7fce0c144ff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:49.499 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 1 -- 172.21.15.148:0/1498382278 >> 172.21.15.148:0/1498382278 conn(0x7fce0c060bf0 msgr2=0x7fce0c061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:49.499 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 1 -- 172.21.15.148:0/1498382278 shutdown_connections 2023-10-30T12:17:49.499 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.455+0000 7fce1371d640 1 -- 172.21.15.148:0/1498382278 wait complete. 2023-10-30T12:17:49.499 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.456+0000 7fce1371d640 10 monclient: build_initial_monmap 2023-10-30T12:17:49.500 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.456+0000 7fce1371d640 10 monclient: monmap: 2023-10-30T12:17:49.500 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:49.500 INFO:teuthology.misc.health.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:49.500 INFO:teuthology.misc.health.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:49.500 INFO:teuthology.misc.health.smithi148.stderr:created 0.000000 2023-10-30T12:17:49.501 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:49.501 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:49.501 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:49.501 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:49.501 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:49.502 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:49.502 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.456+0000 7fce1371d640 1 librados: starting msgr at 2023-10-30T12:17:49.502 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.456+0000 7fce1371d640 1 librados: starting objecter 2023-10-30T12:17:49.502 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.456+0000 7fce1371d640 1 Processor -- start 2023-10-30T12:17:49.502 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.456+0000 7fce1371d640 1 -- start start 2023-10-30T12:17:49.503 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.456+0000 7fce1371d640 1 librados: setting wanted keys 2023-10-30T12:17:49.503 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.456+0000 7fce1371d640 1 librados: calling monclient init 2023-10-30T12:17:49.503 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.456+0000 7fce1371d640 10 monclient: init 2023-10-30T12:17:49.503 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:49.503 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:17:49.503 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce0c0b4590 0x7fce0c1420a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:49.504 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 10 monclient(hunting): picked mon.noname-b con 0x7fce0c0b4590 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:49.504 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c13f1e0 0x7fce0c0ab2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:49.504 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 10 monclient(hunting): picked mon.noname-a con 0x7fce0c13f1e0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:49.504 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c145530 0x7fce0c0ad320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:49.504 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 10 monclient(hunting): picked mon.noname-c con 0x7fce0c145530 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:49.505 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:49.505 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0a8dd0 con 0x7fce0c13f1e0 2023-10-30T12:17:49.505 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:49.505 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c141260 con 0x7fce0c0b4590 2023-10-30T12:17:49.505 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:49.505 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0ace00 con 0x7fce0c145530 2023-10-30T12:17:49.506 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:49.506 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce1371d640 10 monclient(hunting): authenticate will time out at 1561.662095s 2023-10-30T12:17:49.506 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.457+0000 7fce114b9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce0c0b4590 0x7fce0c1420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:49.506 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce114b9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce0c0b4590 0x7fce0c1420a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:36486/0 (socket says 172.21.15.148:36486) 2023-10-30T12:17:49.506 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce114b9640 1 -- 172.21.15.148:0/775334672 learned_addr learned my addr 172.21.15.148:0/775334672 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:49.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce114b9640 10 monclient(hunting): get_auth_request con 0x7fce0c0b4590 auth_method 0 2023-10-30T12:17:49.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce114b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:49.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce114b9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:49.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce114b9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:49.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce11cba640 1 --2- 172.21.15.148:0/775334672 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c145530 0x7fce0c0ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:49.507 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce114b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:49.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce114b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:49.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce114b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:49.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce11cba640 10 monclient(hunting): get_auth_request con 0x7fce0c145530 auth_method 0 2023-10-30T12:17:49.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce11cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:49.508 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce11cba640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:49.509 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.458+0000 7fce11cba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:49.509 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.459+0000 7fce11cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:49.509 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.459+0000 7fce11cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:49.509 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.459+0000 7fce11cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:49.509 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.461+0000 7fce114b9640 10 monclient(hunting): handle_auth_done global_id 4255 payload 824 2023-10-30T12:17:49.509 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.461+0000 7fce114b9640 1 -- 172.21.15.148:0/775334672 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c145530 msgr2=0x7fce0c0ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:49.510 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.461+0000 7fce114b9640 1 --2- 172.21.15.148:0/775334672 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c145530 0x7fce0c0ad320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:49.510 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.461+0000 7fce114b9640 1 -- 172.21.15.148:0/775334672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c13f1e0 msgr2=0x7fce0c0ab2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:49.510 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.461+0000 7fce114b9640 1 --2- 172.21.15.148:0/775334672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c13f1e0 0x7fce0c0ab2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:49.510 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.461+0000 7fce114b9640 10 monclient: _finish_hunting 0 2023-10-30T12:17:49.511 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.461+0000 7fce114b9640 1 monclient: found mon.noname-b 2023-10-30T12:17:49.511 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.461+0000 7fce114b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:49.511 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.461+0000 7fce114b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:49.511 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.461+0000 7fce114b9640 1 -- 172.21.15.148:0/775334672 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c142660 con 0x7fce0c0b4590 2023-10-30T12:17:49.512 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.463+0000 7fce114b9640 1 --2- 172.21.15.148:0/775334672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce0c0b4590 0x7fce0c1420a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc000c00 tx=0x7fcdfc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:49.513 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.463+0000 7fce01ffb640 10 client.?.objecter ms_handle_connect 0x7fce0c0b4590 2023-10-30T12:17:49.513 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.463+0000 7fce01ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:49.513 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 1 -- 172.21.15.148:0/775334672 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc032880 con 0x7fce0c0b4590 2023-10-30T12:17:49.513 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:49.514 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:49.514 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 10 monclient: dump: 2023-10-30T12:17:49.514 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:49.514 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:49.514 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:49.515 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:49.515 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:49.515 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:49.515 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:49.515 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:49.516 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:49.516 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:49.516 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 10 monclient: _finish_auth 0 2023-10-30T12:17:49.516 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:17:49.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:49.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 1 -- 172.21.15.148:0/775334672 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc03f030 con 0x7fce0c0b4590 2023-10-30T12:17:49.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce1371d640 5 monclient: authenticate success, global_id 4255 2023-10-30T12:17:49.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:49.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 1 -- 172.21.15.148:0/775334672 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc032c50 con 0x7fce0c0b4590 2023-10-30T12:17:49.517 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:49.518 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:49.518 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce01ffb640 10 monclient: dump: 2023-10-30T12:17:49.518 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:49.518 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:49.518 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:49.519 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:49.519 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:49.519 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:49.519 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:49.519 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:49.520 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:49.520 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:49.520 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce037fe640 4 set_mon_vals no callback set 2023-10-30T12:17:49.520 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce1371d640 10 monclient: _renew_subs 2023-10-30T12:17:49.520 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce1371d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:49.520 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.466+0000 7fce1371d640 1 -- 172.21.15.148:0/775334672 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c142800 con 0x7fce0c0b4590 2023-10-30T12:17:49.521 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.467+0000 7fce1371d640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:49.521 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.467+0000 7fce1371d640 10 monclient: _renew_subs 2023-10-30T12:17:49.521 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.467+0000 7fce1371d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:49.521 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.467+0000 7fce1371d640 1 -- 172.21.15.148:0/775334672 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c1d6c00 con 0x7fce0c0b4590 2023-10-30T12:17:49.521 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.467+0000 7fce1371d640 1 librados: init done 2023-10-30T12:17:49.521 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.467+0000 7fce1371d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:49.522 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.467+0000 7fce1371d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:49.522 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.467+0000 7fce1371d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:49.522 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.467+0000 7fce1371d640 1 -- 172.21.15.148:0/775334672 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c0ae0d0 con 0x7fce0c0b4590 2023-10-30T12:17:49.714 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.711+0000 7fce01ffb640 1 -- 172.21.15.148:0/775334672 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fcdfc0443a0 con 0x7fce0c0b4590 2023-10-30T12:17:49.714 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.711+0000 7fce01ffb640 1 --2- 172.21.15.148:0/775334672 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcdf006c780 0x7fcdf006ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:49.715 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.712+0000 7fce10cb8640 1 --2- 172.21.15.148:0/775334672 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcdf006c780 0x7fcdf006ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:49.715 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.712+0000 7fce10cb8640 10 monclient: get_auth_request con 0x7fcdf006c780 auth_method 0 2023-10-30T12:17:49.715 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.712+0000 7fce10cb8640 1 --2- 172.21.15.148:0/775334672 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcdf006c780 0x7fcdf006ec40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce0c1d47f0 tx=0x7fce08009500 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:49.716 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.712+0000 7fce01ffb640 10 client.4255.objecter ms_handle_connect 0x7fcdf006c780 2023-10-30T12:17:49.725 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.723+0000 7fce01ffb640 1 -- 172.21.15.148:0/775334672 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 1988+0+0 (secure 0 0 0) 0x7fcdfc0795c0 con 0x7fce0c0b4590 2023-10-30T12:17:49.725 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.723+0000 7fce01ffb640 10 client.4255.objecter ms_dispatch 0x7fce0c000b90 osd_map(12..12 src has 1..12) v4 2023-10-30T12:17:49.725 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.723+0000 7fce01ffb640 3 client.4255.objecter handle_osd_map got epochs [12,12] > 0 2023-10-30T12:17:49.726 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.723+0000 7fce01ffb640 3 client.4255.objecter handle_osd_map decoding full epoch 12 2023-10-30T12:17:49.726 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.723+0000 7fce01ffb640 20 client.4255.objecter dump_active .. 0 homeless 2023-10-30T12:17:49.727 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.724+0000 7fce01ffb640 1 -- 172.21.15.148:0/775334672 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcdfc037070 con 0x7fce0c0b4590 2023-10-30T12:17:49.727 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.724+0000 7fce01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:49.727 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.724+0000 7fce01ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:49.935 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.934+0000 7fce1371d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:49.936 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.934+0000 7fce1371d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:49.936 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.934+0000 7fce1371d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:49.936 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:49.934+0000 7fce1371d640 1 -- 172.21.15.148:0/775334672 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce0c0ae0d0 con 0x7fce0c0b4590 2023-10-30T12:17:50.052 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:17:50.047+0000 c495640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T12:17:50.090 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.089+0000 7fce01ffb640 1 -- 172.21.15.148:0/775334672 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+7322 (secure 0 0 0) 0x7fcdfc078ec0 con 0x7fce0c0b4590 2023-10-30T12:17:50.091 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.089+0000 7fce01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:50.091 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.089+0000 7fce01ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:50.091 INFO:teuthology.misc.health.smithi148.stdout: 2023-10-30T12:17:50.091 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":13,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:17:49.278721+0000","last_up_change":"2023-10-30T12:17:49.278721+0000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2604225934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6801","nonce":2604225934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":2604225934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2604225934}]},"public_addr":"172.21.15.150:6800/2604225934","cluster_addr":"172.21.15.150:6801/2604225934","heartbeat_back_addr":"172.21.15.150:6803/2604225934","heartbeat_front_addr":"172.21.15.150:6802/2604225934","state":["exists","up"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:17:50.094 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 10 librados: watch_flush enter 2023-10-30T12:17:50.094 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 10 librados: watch_flush exit 2023-10-30T12:17:50.094 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 20 client.4255.objecter shutdown clearing up homeless session... 2023-10-30T12:17:50.095 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 10 client.4255.objecter successfully canceled tick 2023-10-30T12:17:50.095 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 -- 172.21.15.148:0/775334672 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcdf006c780 msgr2=0x7fcdf006ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:50.095 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 --2- 172.21.15.148:0/775334672 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcdf006c780 0x7fcdf006ec40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce0c1d47f0 tx=0x7fce08009500 comp rx=0 tx=0).stop 2023-10-30T12:17:50.095 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 10 monclient: shutdown 2023-10-30T12:17:50.095 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:50.096 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 -- 172.21.15.148:0/775334672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce0c0b4590 msgr2=0x7fce0c1420a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:50.096 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 --2- 172.21.15.148:0/775334672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce0c0b4590 0x7fce0c1420a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc000c00 tx=0x7fcdfc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:17:50.096 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 -- 172.21.15.148:0/775334672 shutdown_connections 2023-10-30T12:17:50.096 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 --2- 172.21.15.148:0/775334672 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcdf006c780 0x7fcdf006ec40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:50.096 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 --2- 172.21.15.148:0/775334672 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce0c145530 0x7fce0c0ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:50.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 --2- 172.21.15.148:0/775334672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce0c13f1e0 0x7fce0c0ab2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:50.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 --2- 172.21.15.148:0/775334672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce0c0b4590 0x7fce0c1420a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:50.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 -- 172.21.15.148:0/775334672 >> 172.21.15.148:0/775334672 conn(0x7fce0c060bf0 msgr2=0x7fce0c0b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:50.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 -- 172.21.15.148:0/775334672 shutdown_connections 2023-10-30T12:17:50.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 -- 172.21.15.148:0/775334672 wait complete. 2023-10-30T12:17:50.097 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:50.092+0000 7fce1371d640 1 librados: shutdown 2023-10-30T12:17:50.103 DEBUG:teuthology.misc:1 of 8 OSDs are up 2023-10-30T12:17:50.115 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T12:17:50.111+0000 ac7d640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T12:17:50.159 INFO:tasks.ceph.osd.7.smithi150.stderr:2023-10-30T12:17:50.154+0000 c495640 -1 osd.7 0 waiting for initial osdmap 2023-10-30T12:17:50.180 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T12:17:50.175+0000 c495640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T12:17:50.196 INFO:tasks.ceph.osd.4.smithi150.stderr:2023-10-30T12:17:50.191+0000 c495640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T12:17:50.225 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:17:50.220+0000 ac7d640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T12:17:50.495 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:17:50.481+0000 11a5d640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:17:50.576 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T12:17:50.561+0000 11a5d640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:17:50.630 INFO:tasks.ceph.osd.7.smithi150.stderr:2023-10-30T12:17:50.611+0000 11a5d640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:17:50.638 INFO:tasks.ceph.osd.4.smithi150.stderr:2023-10-30T12:17:50.625+0000 1225e640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:17:50.649 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T12:17:50.629+0000 1225e640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:17:50.680 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:17:50.665+0000 1225e640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:17:50.792 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T12:17:50.786+0000 ac7d640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T12:17:51.246 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T12:17:51.231+0000 11a5d640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:17:56.911 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:17:57.005 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.003+0000 7fedd6d05640 10 monclient: get_monmap_and_config 2023-10-30T12:17:57.005 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.003+0000 7fedd6d05640 10 monclient: build_initial_monmap 2023-10-30T12:17:57.005 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.003+0000 7fedd6d05640 10 monclient: monmap: 2023-10-30T12:17:57.005 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:57.006 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:57.006 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:17:57.004553+0000 2023-10-30T12:17:57.006 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:17:57.004553+0000 2023-10-30T12:17:57.006 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:57.006 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:57.007 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:57.007 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:57.007 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:57.007 INFO:teuthology.misc.health.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:57.007 INFO:teuthology.misc.health.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:57.008 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:57.009 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 1 Processor -- start 2023-10-30T12:17:57.010 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 1 -- start start 2023-10-30T12:17:57.010 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 10 monclient: init 2023-10-30T12:17:57.010 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:57.010 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-10-30T12:17:57.011 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fedd00b4590 0x7fedd0144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.011 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 10 monclient(hunting): picked mon.noname-a con 0x7fedd00b4590 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:57.011 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fedd0145530 0x7fedd013eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.011 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 10 monclient(hunting): picked mon.noname-c con 0x7fedd0145530 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:57.011 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fedd013f1e0 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:57.011 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd00aa540 con 0x7fedd013f1e0 2023-10-30T12:17:57.012 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:57.012 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd0067160 con 0x7fedd00b4590 2023-10-30T12:17:57.012 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:57.012 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd01459c0 con 0x7fedd0145530 2023-10-30T12:17:57.012 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:57.013 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.004+0000 7fedd6d05640 10 monclient(hunting): authenticate will time out at 1569.209743s 2023-10-30T12:17:57.013 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.006+0000 7fedcffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fedd0145530 0x7fedd013eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:57.013 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.006+0000 7fedcffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fedd0145530 0x7fedd013eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:57710/0 (socket says 172.21.15.148:57710) 2023-10-30T12:17:57.013 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.006+0000 7fedcffff640 1 -- 172.21.15.148:0/3749229797 learned_addr learned my addr 172.21.15.148:0/3749229797 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:57.013 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.006+0000 7fedcffff640 10 monclient(hunting): get_auth_request con 0x7fedd0145530 auth_method 0 2023-10-30T12:17:57.013 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.006+0000 7fedcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:57.014 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.006+0000 7fedcffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:57.014 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.006+0000 7fedcffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:57.014 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.007+0000 7fedcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:57.014 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.007+0000 7fedcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:57.014 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.007+0000 7fedcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:57.015 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.009+0000 7fedcffff640 10 monclient(hunting): handle_auth_done global_id 4258 payload 274 2023-10-30T12:17:57.015 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.009+0000 7fedcffff640 1 -- 172.21.15.148:0/3749229797 >> v2:172.21.15.148:3300/0 conn(0x7fedd00b4590 msgr2=0x7fedd0144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:57.016 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.009+0000 7fedcffff640 1 --2- 172.21.15.148:0/3749229797 >> v2:172.21.15.148:3300/0 conn(0x7fedd00b4590 0x7fedd0144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.016 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.009+0000 7fedcffff640 1 -- 172.21.15.148:0/3749229797 >> v1:172.21.15.150:6789/0 conn(0x7fedd013f1e0 legacy=0x7fedd00b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.016 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.009+0000 7fedcffff640 10 monclient: _finish_hunting 0 2023-10-30T12:17:57.016 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.009+0000 7fedcffff640 1 monclient: found mon.noname-c 2023-10-30T12:17:57.016 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.009+0000 7fedcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:57.017 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.009+0000 7fedcffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.017 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.009+0000 7fedcffff640 1 -- 172.21.15.148:0/3749229797 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd00aa7b0 con 0x7fedd0145530 2023-10-30T12:17:57.017 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.013+0000 7fedcffff640 1 --2- 172.21.15.148:0/3749229797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd0145530 0x7fedd013eca0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fedd00b4ec0 tx=0x7fedbc02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d9c9fb4750150c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:57.018 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 1 -- 172.21.15.148:0/3749229797 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc031870 con 0x7fedd0145530 2023-10-30T12:17:57.019 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:57.019 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:57.019 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 10 monclient: dump: 2023-10-30T12:17:57.019 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:57.020 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:57.020 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.020 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.020 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:57.020 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:57.021 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:57.021 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:57.021 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:57.021 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:57.021 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:57.021 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:57.022 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:57.022 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 1 -- 172.21.15.148:0/3749229797 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedbc0319e0 con 0x7fedd0145530 2023-10-30T12:17:57.022 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:57.022 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 1 -- 172.21.15.148:0/3749229797 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc031d80 con 0x7fedd0145530 2023-10-30T12:17:57.022 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:57.023 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:57.023 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedcf7fe640 10 monclient: dump: 2023-10-30T12:17:57.023 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:57.023 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:57.023 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.024 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.024 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:57.024 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:57.024 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:57.024 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:57.025 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:57.025 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:57.025 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedd6d05640 5 monclient: authenticate success, global_id 4258 2023-10-30T12:17:57.025 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedd6d05640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:57.025 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.016+0000 7fedd6d05640 4 set_mon_vals no callback set 2023-10-30T12:17:57.025 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.017+0000 7fedd6d05640 10 monclient: shutdown 2023-10-30T12:17:57.026 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.017+0000 7fedd6d05640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:57.026 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.017+0000 7fedd6d05640 1 -- 172.21.15.148:0/3749229797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd0145530 msgr2=0x7fedd013eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.026 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.017+0000 7fedd6d05640 1 --2- 172.21.15.148:0/3749229797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd0145530 0x7fedd013eca0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fedd00b4ec0 tx=0x7fedbc02b910 comp rx=0 tx=0).stop 2023-10-30T12:17:57.026 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.017+0000 7fedd6d05640 1 -- 172.21.15.148:0/3749229797 shutdown_connections 2023-10-30T12:17:57.027 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.017+0000 7fedd6d05640 1 --2- 172.21.15.148:0/3749229797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd0145530 0x7fedd013eca0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.027 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.017+0000 7fedd6d05640 1 --2- 172.21.15.148:0/3749229797 >> v2:172.21.15.148:3300/0 conn(0x7fedd00b4590 0x7fedd0144ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.027 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.017+0000 7fedd6d05640 1 -- 172.21.15.148:0/3749229797 >> 172.21.15.148:0/3749229797 conn(0x7fedd0060bf0 msgr2=0x7fedd0061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:57.027 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.017+0000 7fedd6d05640 1 -- 172.21.15.148:0/3749229797 shutdown_connections 2023-10-30T12:17:57.027 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.017+0000 7fedd6d05640 1 -- 172.21.15.148:0/3749229797 wait complete. 2023-10-30T12:17:57.029 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.018+0000 7fedd6d05640 10 monclient: build_initial_monmap 2023-10-30T12:17:57.030 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.018+0000 7fedd6d05640 10 monclient: monmap: 2023-10-30T12:17:57.030 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2023-10-30T12:17:57.030 INFO:teuthology.misc.health.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:57.030 INFO:teuthology.misc.health.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:57.030 INFO:teuthology.misc.health.smithi148.stderr:created 0.000000 2023-10-30T12:17:57.031 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:57.031 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:57.031 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:57.031 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:57.031 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:57.032 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:57.032 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.018+0000 7fedd6d05640 1 librados: starting msgr at 2023-10-30T12:17:57.032 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.018+0000 7fedd6d05640 1 librados: starting objecter 2023-10-30T12:17:57.032 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.018+0000 7fedd6d05640 1 Processor -- start 2023-10-30T12:17:57.032 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.018+0000 7fedd6d05640 1 -- start start 2023-10-30T12:17:57.032 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.018+0000 7fedd6d05640 1 librados: setting wanted keys 2023-10-30T12:17:57.033 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.018+0000 7fedd6d05640 1 librados: calling monclient init 2023-10-30T12:17:57.033 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.018+0000 7fedd6d05640 10 monclient: init 2023-10-30T12:17:57.033 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:57.033 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:17:57.033 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fedd00b4590 0x7fedd013fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.034 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 10 monclient(hunting): picked mon.noname-c con 0x7fedd00b4590 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:57.034 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd013f1e0 0x7fedd00ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.034 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 10 monclient(hunting): picked mon.noname-b con 0x7fedd013f1e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:57.034 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fedd0145530 0x7fedd01417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.034 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 10 monclient(hunting): picked mon.noname-a con 0x7fedd0145530 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:57.035 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:57.035 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd00a8dd0 con 0x7fedd0145530 2023-10-30T12:17:57.035 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:57.035 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd01d9fb0 con 0x7fedd013f1e0 2023-10-30T12:17:57.035 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:57.035 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd01d9c60 con 0x7fedd00b4590 2023-10-30T12:17:57.036 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:57.036 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedd6d05640 10 monclient(hunting): authenticate will time out at 1569.224211s 2023-10-30T12:17:57.036 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.019+0000 7fedcffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd013f1e0 0x7fedd00ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:57.036 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.020+0000 7fedcffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd013f1e0 0x7fedd00ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:57720/0 (socket says 172.21.15.148:57720) 2023-10-30T12:17:57.036 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.020+0000 7fedcffff640 1 -- 172.21.15.148:0/3078944968 learned_addr learned my addr 172.21.15.148:0/3078944968 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:57.037 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.020+0000 7fedcffff640 10 monclient(hunting): get_auth_request con 0x7fedd013f1e0 auth_method 0 2023-10-30T12:17:57.037 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.020+0000 7fedcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:57.037 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.020+0000 7fedcffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:57.037 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.020+0000 7fedcffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:57.037 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.020+0000 7fedcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:57.038 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.020+0000 7fedcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:57.038 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.020+0000 7fedcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:57.038 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.023+0000 7fedcffff640 10 monclient(hunting): handle_auth_done global_id 4261 payload 824 2023-10-30T12:17:57.038 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.023+0000 7fedcffff640 1 -- 172.21.15.148:0/3078944968 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fedd00b4590 msgr2=0x7fedd013fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.038 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.023+0000 7fedcffff640 1 --2- 172.21.15.148:0/3078944968 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fedd00b4590 0x7fedd013fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.039 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.023+0000 7fedcffff640 1 -- 172.21.15.148:0/3078944968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fedd0145530 msgr2=0x7fedd01417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.039 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.023+0000 7fedcffff640 1 --2- 172.21.15.148:0/3078944968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fedd0145530 0x7fedd01417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.039 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.023+0000 7fedcffff640 10 monclient: _finish_hunting 0 2023-10-30T12:17:57.039 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.023+0000 7fedcffff640 1 monclient: found mon.noname-b 2023-10-30T12:17:57.039 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.023+0000 7fedcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:57.040 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.023+0000 7fedcffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.040 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.023+0000 7fedcffff640 1 -- 172.21.15.148:0/3078944968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd0140130 con 0x7fedd013f1e0 2023-10-30T12:17:57.040 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.025+0000 7fedcffff640 1 --2- 172.21.15.148:0/3078944968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd013f1e0 0x7fedd00ad1c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fedbc000c00 tx=0x7fedbc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:57.040 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.025+0000 7fedcd7fa640 10 client.?.objecter ms_handle_connect 0x7fedd013f1e0 2023-10-30T12:17:57.040 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.025+0000 7fedcd7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:57.042 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 1 -- 172.21.15.148:0/3078944968 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc03adf0 con 0x7fedd013f1e0 2023-10-30T12:17:57.042 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:57.042 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:57.042 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 10 monclient: dump: 2023-10-30T12:17:57.043 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:57.043 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:57.043 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.043 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.043 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:57.044 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:57.044 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:57.044 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:57.044 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:57.044 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:57.044 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 10 monclient: _finish_auth 0 2023-10-30T12:17:57.045 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:17:57.045 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:57.045 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedd6d05640 5 monclient: authenticate success, global_id 4261 2023-10-30T12:17:57.045 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 1 -- 172.21.15.148:0/3078944968 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedbc03e030 con 0x7fedd013f1e0 2023-10-30T12:17:57.045 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedd6d05640 10 monclient: _renew_subs 2023-10-30T12:17:57.046 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedd6d05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.046 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedd6d05640 1 -- 172.21.15.148:0/3078944968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd0140330 con 0x7fedd013f1e0 2023-10-30T12:17:57.046 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:57.046 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 1 -- 172.21.15.148:0/3078944968 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc044630 con 0x7fedd013f1e0 2023-10-30T12:17:57.046 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:57.047 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:57.047 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedcd7fa640 10 monclient: dump: 2023-10-30T12:17:57.047 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2023-10-30T12:17:57.047 INFO:teuthology.misc.health.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:57.047 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.047 INFO:teuthology.misc.health.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.048 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:57.048 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:57.048 INFO:teuthology.misc.health.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:57.048 INFO:teuthology.misc.health.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:57.048 INFO:teuthology.misc.health.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:57.049 INFO:teuthology.misc.health.smithi148.stderr: 2023-10-30T12:17:57.049 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedceffd640 4 set_mon_vals no callback set 2023-10-30T12:17:57.049 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedd6d05640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:57.049 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedd6d05640 10 monclient: _renew_subs 2023-10-30T12:17:57.049 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedd6d05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.049 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedd6d05640 1 -- 172.21.15.148:0/3078944968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd01da6b0 con 0x7fedd013f1e0 2023-10-30T12:17:57.050 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.029+0000 7fedd6d05640 1 librados: init done 2023-10-30T12:17:57.050 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.030+0000 7fedd6d05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:57.050 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.030+0000 7fedd6d05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:57.050 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.030+0000 7fedd6d05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.050 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.030+0000 7fedd6d05640 1 -- 172.21.15.148:0/3078944968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd0075550 con 0x7fedd013f1e0 2023-10-30T12:17:57.052 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.047+0000 7fedcd7fa640 1 -- 172.21.15.148:0/3078944968 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fedbc04d450 con 0x7fedd013f1e0 2023-10-30T12:17:57.052 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.047+0000 7fedcd7fa640 1 --2- 172.21.15.148:0/3078944968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fedb406c780 0x7fedb406ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.052 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.047+0000 7fedd4aa1640 1 --2- 172.21.15.148:0/3078944968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fedb406c780 0x7fedb406ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:57.052 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.048+0000 7fedd4aa1640 10 monclient: get_auth_request con 0x7fedb406c780 auth_method 0 2023-10-30T12:17:57.052 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.048+0000 7fedcd7fa640 10 client.4261.objecter ms_handle_connect 0x7fedb406c780 2023-10-30T12:17:57.053 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.048+0000 7fedd4aa1640 1 --2- 172.21.15.148:0/3078944968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fedb406c780 0x7fedb406ec40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009710 tx=0x7fedc4006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:57.053 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.049+0000 7fedcd7fa640 1 -- 172.21.15.148:0/3078944968 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3737+0+0 (secure 0 0 0) 0x7fedbc07a990 con 0x7fedd013f1e0 2023-10-30T12:17:57.053 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.049+0000 7fedcd7fa640 10 client.4261.objecter ms_dispatch 0x7fedd0000b90 osd_map(19..19 src has 1..19) v4 2023-10-30T12:17:57.053 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.049+0000 7fedcd7fa640 3 client.4261.objecter handle_osd_map got epochs [19,19] > 0 2023-10-30T12:17:57.053 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.049+0000 7fedcd7fa640 3 client.4261.objecter handle_osd_map decoding full epoch 19 2023-10-30T12:17:57.053 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.049+0000 7fedcd7fa640 20 client.4261.objecter dump_active .. 0 homeless 2023-10-30T12:17:57.306 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.305+0000 7fedcd7fa640 1 -- 172.21.15.148:0/3078944968 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fedbc07a290 con 0x7fedd013f1e0 2023-10-30T12:17:57.306 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.305+0000 7fedcd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:57.307 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.305+0000 7fedcd7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:57.537 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.536+0000 7fedd6d05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:57.538 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.536+0000 7fedd6d05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:57.538 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.536+0000 7fedd6d05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.538 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.536+0000 7fedd6d05640 1 -- 172.21.15.148:0/3078944968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedd00aa7b0 con 0x7fedd013f1e0 2023-10-30T12:17:57.623 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.621+0000 7fedcd7fa640 1 -- 172.21.15.148:0/3078944968 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+10950 (secure 0 0 0) 0x7fedbc07e3e0 con 0x7fedd013f1e0 2023-10-30T12:17:57.623 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.621+0000 7fedcd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:17:57.623 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.621+0000 7fedcd7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:57.624 INFO:teuthology.misc.health.smithi148.stdout: 2023-10-30T12:17:57.624 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":20,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:17:57.037012+0000","last_up_change":"2023-10-30T12:17:51.437771+0000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:17:53.315604+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":2395694124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2395694124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2395694124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2395694124}]},"public_addr":"172.21.15.148:6805/2395694124","cluster_addr":"172.21.15.148:6807/2395694124","heartbeat_back_addr":"172.21.15.148:6810/2395694124","heartbeat_front_addr":"172.21.15.148:6808/2395694124","state":["exists","up"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":912412250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":912412250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":912412250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":912412250}]},"public_addr":"172.21.15.148:6813/912412250","cluster_addr":"172.21.15.148:6814/912412250","heartbeat_back_addr":"172.21.15.148:6816/912412250","heartbeat_front_addr":"172.21.15.148:6815/912412250","state":["exists","up"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":2368806934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2368806934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":2368806934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":2368806934}]},"public_addr":"172.21.15.148:6801/2368806934","cluster_addr":"172.21.15.148:6802/2368806934","heartbeat_back_addr":"172.21.15.148:6804/2368806934","heartbeat_front_addr":"172.21.15.148:6803/2368806934","state":["exists","up"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":2233349848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":2233349848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2233349848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":2233349848}]},"public_addr":"172.21.15.148:6806/2233349848","cluster_addr":"172.21.15.148:6809/2233349848","heartbeat_back_addr":"172.21.15.148:6812/2233349848","heartbeat_front_addr":"172.21.15.148:6811/2233349848","state":["exists","up"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2836839739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":2836839739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":2836839739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2836839739}]},"public_addr":"172.21.15.150:6808/2836839739","cluster_addr":"172.21.15.150:6809/2836839739","heartbeat_back_addr":"172.21.15.150:6811/2836839739","heartbeat_front_addr":"172.21.15.150:6810/2836839739","state":["exists","up"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4190376848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":4190376848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6807","nonce":4190376848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4190376848}]},"public_addr":"172.21.15.150:6804/4190376848","cluster_addr":"172.21.15.150:6805/4190376848","heartbeat_back_addr":"172.21.15.150:6807/4190376848","heartbeat_front_addr":"172.21.15.150:6806/4190376848","state":["exists","up"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2604225934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6801","nonce":2604225934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":2604225934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2604225934}]},"public_addr":"172.21.15.150:6800/2604225934","cluster_addr":"172.21.15.150:6801/2604225934","heartbeat_back_addr":"172.21.15.150:6803/2604225934","heartbeat_front_addr":"172.21.15.150:6802/2604225934","state":["exists","up"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3463178370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":3463178370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":3463178370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3463178370}]},"public_addr":"172.21.15.150:6812/3463178370","cluster_addr":"172.21.15.150:6813/3463178370","heartbeat_back_addr":"172.21.15.150:6815/3463178370","heartbeat_front_addr":"172.21.15.150:6814/3463178370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.920964+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.821114+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.365893+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:49.037497+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.184019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.644370+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:47.688638+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.429422+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:17:57.626 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 10 librados: watch_flush enter 2023-10-30T12:17:57.626 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 10 librados: watch_flush exit 2023-10-30T12:17:57.626 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 20 client.4261.objecter shutdown clearing up homeless session... 2023-10-30T12:17:57.627 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 10 client.4261.objecter successfully canceled tick 2023-10-30T12:17:57.627 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 -- 172.21.15.148:0/3078944968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fedb406c780 msgr2=0x7fedb406ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.627 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 --2- 172.21.15.148:0/3078944968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fedb406c780 0x7fedb406ec40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009710 tx=0x7fedc4006c90 comp rx=0 tx=0).stop 2023-10-30T12:17:57.627 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 10 monclient: shutdown 2023-10-30T12:17:57.627 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:57.628 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 -- 172.21.15.148:0/3078944968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd013f1e0 msgr2=0x7fedd00ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.628 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 --2- 172.21.15.148:0/3078944968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd013f1e0 0x7fedd00ad1c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fedbc000c00 tx=0x7fedbc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.628 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 -- 172.21.15.148:0/3078944968 shutdown_connections 2023-10-30T12:17:57.628 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 --2- 172.21.15.148:0/3078944968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fedb406c780 0x7fedb406ec40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.628 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 --2- 172.21.15.148:0/3078944968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fedd0145530 0x7fedd01417c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.628 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 --2- 172.21.15.148:0/3078944968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd013f1e0 0x7fedd00ad1c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.629 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 --2- 172.21.15.148:0/3078944968 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fedd00b4590 0x7fedd013fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.629 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 -- 172.21.15.148:0/3078944968 >> 172.21.15.148:0/3078944968 conn(0x7fedd0060bf0 msgr2=0x7fedd00b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:57.629 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 -- 172.21.15.148:0/3078944968 shutdown_connections 2023-10-30T12:17:57.629 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 -- 172.21.15.148:0/3078944968 wait complete. 2023-10-30T12:17:57.629 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T12:17:57.624+0000 7fedd6d05640 1 librados: shutdown 2023-10-30T12:17:57.635 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-30T12:17:57.635 INFO:tasks.ceph:Creating RBD pool 2023-10-30T12:17:57.636 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-30T12:17:57.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.738+0000 7fb7058e0640 10 monclient: get_monmap_and_config 2023-10-30T12:17:57.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.738+0000 7fb7058e0640 10 monclient: build_initial_monmap 2023-10-30T12:17:57.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.738+0000 7fb7058e0640 10 monclient: monmap: 2023-10-30T12:17:57.740 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:17:57.740 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:57.741 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:17:57.739517+0000 2023-10-30T12:17:57.741 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:17:57.739517+0000 2023-10-30T12:17:57.741 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:57.741 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:57.741 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:57.742 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:57.742 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:57.742 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:57.742 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:57.742 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:57.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 1 Processor -- start 2023-10-30T12:17:57.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 1 -- start start 2023-10-30T12:17:57.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 10 monclient: init 2023-10-30T12:17:57.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:57.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-10-30T12:17:57.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7000b7c10 addr v1:172.21.15.148:6789/0 2023-10-30T12:17:57.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb700141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:17:57.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb70014a460 0x7fb70014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 10 monclient(hunting): picked mon.noname-c con 0x7fb70014a460 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:57.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7000a9350 con 0x7fb7000b7c10 2023-10-30T12:17:57.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7000aa6a0 con 0x7fb700141940 2023-10-30T12:17:57.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:57.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7000670a0 con 0x7fb70014a460 2023-10-30T12:17:57.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:57.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.739+0000 7fb7058e0640 10 monclient(hunting): authenticate will time out at 1569.944701s 2023-10-30T12:17:57.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.740+0000 7fb6ff7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb70014a460 0x7fb70014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:57.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.740+0000 7fb6feffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb7000b7c10 0x7fb7000a8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58994/0 (socket says 172.21.15.148:58994) 2023-10-30T12:17:57.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.740+0000 7fb6feffd640 1 -- 172.21.15.148:0/2156034677 learned_addr learned my addr 172.21.15.148:0/2156034677 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:17:57.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.740+0000 7fb6ff7fe640 10 monclient(hunting): get_auth_request con 0x7fb70014a460 auth_method 0 2023-10-30T12:17:57.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.740+0000 7fb6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:57.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.740+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:57.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.740+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:57.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.741+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:57.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.741+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:57.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.741+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:57.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6fe7fc640 1 -- 172.21.15.148:0/2156034677 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645995907 0 0) 0x7fb6f0001110 con 0x7fb7000b7c10 2023-10-30T12:17:57.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6fe7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:17:57.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6fe7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:57.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6fe7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:57.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6fe7fc640 10 monclient(hunting): my global_id is 4248 2023-10-30T12:17:57.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6fe7fc640 1 -- 172.21.15.148:0/2156034677 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6ec003520 con 0x7fb7000b7c10 2023-10-30T12:17:57.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_done global_id 4264 payload 274 2023-10-30T12:17:57.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6ff7fe640 1 -- 172.21.15.148:0/2156034677 >> v1:172.21.15.150:6789/0 conn(0x7fb700141940 legacy=0x7fb700141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6ff7fe640 1 -- 172.21.15.148:0/2156034677 >> v1:172.21.15.148:6789/0 conn(0x7fb7000b7c10 legacy=0x7fb7000a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6ff7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:17:57.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6ff7fe640 1 monclient: found mon.noname-c 2023-10-30T12:17:57.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:57.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.743+0000 7fb6ff7fe640 1 -- 172.21.15.148:0/2156034677 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7000aa910 con 0x7fb70014a460 2023-10-30T12:17:57.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.748+0000 7fb6ff7fe640 1 --2- 172.21.15.148:0/2156034677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb70014a460 0x7fb70014c850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009d90 tx=0x7fb6f802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37d5bf8804ccb1e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:57.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.750+0000 7fb6fe7fc640 1 -- 172.21.15.148:0/2156034677 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f802cc60 con 0x7fb70014a460 2023-10-30T12:17:57.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.750+0000 7fb6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:57.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.750+0000 7fb6fe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:57.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.750+0000 7fb6fe7fc640 10 monclient: dump: 2023-10-30T12:17:57.754 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:57.754 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:57.755 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.755 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.755 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:57.755 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:57.755 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:57.755 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:57.756 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:57.756 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:57.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.750+0000 7fb6fe7fc640 10 monclient: _finish_auth 0 2023-10-30T12:17:57.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.750+0000 7fb6fe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:17:57.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.750+0000 7fb6fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:57.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.750+0000 7fb7058e0640 5 monclient: authenticate success, global_id 4264 2023-10-30T12:17:57.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.750+0000 7fb7058e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:57.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.751+0000 7fb6fe7fc640 1 -- 172.21.15.148:0/2156034677 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f802cdd0 con 0x7fb70014a460 2023-10-30T12:17:57.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.751+0000 7fb6fe7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:57.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.751+0000 7fb6fe7fc640 1 -- 172.21.15.148:0/2156034677 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8034640 con 0x7fb70014a460 2023-10-30T12:17:57.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.751+0000 7fb6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:57.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.751+0000 7fb6fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:57.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.751+0000 7fb6fe7fc640 10 monclient: dump: 2023-10-30T12:17:57.758 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:57.758 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:57.758 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.759 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.759 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:57.759 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:57.759 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:57.759 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:57.759 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:57.760 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:57.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.751+0000 7fb7058e0640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:57.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.751+0000 7fb7058e0640 4 set_mon_vals no callback set 2023-10-30T12:17:57.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.752+0000 7fb7058e0640 10 monclient: shutdown 2023-10-30T12:17:57.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.752+0000 7fb7058e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:57.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.752+0000 7fb7058e0640 1 -- 172.21.15.148:0/2156034677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb70014a460 msgr2=0x7fb70014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.752+0000 7fb7058e0640 1 --2- 172.21.15.148:0/2156034677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb70014a460 0x7fb70014c850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009d90 tx=0x7fb6f802beb0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.752+0000 7fb7058e0640 1 -- 172.21.15.148:0/2156034677 shutdown_connections 2023-10-30T12:17:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.752+0000 7fb7058e0640 1 --2- 172.21.15.148:0/2156034677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb70014a460 0x7fb70014c850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.752+0000 7fb7058e0640 1 -- 172.21.15.148:0/2156034677 >> 172.21.15.148:0/2156034677 conn(0x7fb70006c660 msgr2=0x7fb7000b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.752+0000 7fb7058e0640 1 -- 172.21.15.148:0/2156034677 shutdown_connections 2023-10-30T12:17:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.752+0000 7fb7058e0640 1 -- 172.21.15.148:0/2156034677 wait complete. 2023-10-30T12:17:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.753+0000 7fb7058e0640 10 monclient: build_initial_monmap 2023-10-30T12:17:57.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.753+0000 7fb7058e0640 10 monclient: monmap: 2023-10-30T12:17:57.766 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:17:57.766 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:57.766 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:57.766 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:17:57.767 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:57.767 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:57.767 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:57.767 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:57.767 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:57.767 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.753+0000 7fb7058e0640 1 librados: starting msgr at 2023-10-30T12:17:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.753+0000 7fb7058e0640 1 librados: starting objecter 2023-10-30T12:17:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.753+0000 7fb7058e0640 1 Processor -- start 2023-10-30T12:17:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.753+0000 7fb7058e0640 1 -- start start 2023-10-30T12:17:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.753+0000 7fb7058e0640 1 librados: setting wanted keys 2023-10-30T12:17:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.753+0000 7fb7058e0640 1 librados: calling monclient init 2023-10-30T12:17:57.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.753+0000 7fb7058e0640 10 monclient: init 2023-10-30T12:17:57.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:57.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:17:57.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7000b7c10 0x7fb7000ae850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 10 monclient(hunting): picked mon.noname-b con 0x7fb7000b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:57.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb700141940 0x7fb7000ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 10 monclient(hunting): picked mon.noname-a con 0x7fb700141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:57.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb70014a460 0x7fb7000aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 10 monclient(hunting): picked mon.noname-c con 0x7fb70014a460 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:57.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:57.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7000b32f0 con 0x7fb700141940 2023-10-30T12:17:57.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:57.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7000ae3c0 con 0x7fb7000b7c10 2023-10-30T12:17:57.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:57.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7000af730 con 0x7fb70014a460 2023-10-30T12:17:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb7058e0640 10 monclient(hunting): authenticate will time out at 1569.959282s 2023-10-30T12:17:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.754+0000 7fb6f7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb700141940 0x7fb7000ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6feffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7000b7c10 0x7fb7000ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6ff7fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb70014a460 0x7fb7000aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:57.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6f7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb700141940 0x7fb7000ad890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32880/0 (socket says 172.21.15.148:32880) 2023-10-30T12:17:57.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6f7fff640 1 -- 172.21.15.148:0/429013307 learned_addr learned my addr 172.21.15.148:0/429013307 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:57.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6f7fff640 10 monclient(hunting): get_auth_request con 0x7fb700141940 auth_method 0 2023-10-30T12:17:57.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:57.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6f7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:57.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6f7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:57.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6feffd640 10 monclient(hunting): get_auth_request con 0x7fb7000b7c10 auth_method 0 2023-10-30T12:17:57.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:57.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6feffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:57.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6feffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:57.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6ff7fe640 10 monclient(hunting): get_auth_request con 0x7fb70014a460 auth_method 0 2023-10-30T12:17:57.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:57.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:57.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:57.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:57.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:57.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:57.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:57.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:57.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.755+0000 7fb6f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:57.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.756+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:57.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.756+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:57.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.756+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:57.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.758+0000 7fb6feffd640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2023-10-30T12:17:57.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.758+0000 7fb6feffd640 1 -- 172.21.15.148:0/429013307 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb70014a460 msgr2=0x7fb7000aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.758+0000 7fb6feffd640 1 --2- 172.21.15.148:0/429013307 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb70014a460 0x7fb7000aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.758+0000 7fb6feffd640 1 -- 172.21.15.148:0/429013307 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb700141940 msgr2=0x7fb7000ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:57.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.758+0000 7fb6feffd640 1 --2- 172.21.15.148:0/429013307 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb700141940 0x7fb7000ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:57.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.758+0000 7fb6feffd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:57.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.758+0000 7fb6feffd640 1 monclient: found mon.noname-b 2023-10-30T12:17:57.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.758+0000 7fb6feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:57.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.758+0000 7fb6feffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.758+0000 7fb6feffd640 1 -- 172.21.15.148:0/429013307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7000afa10 con 0x7fb7000b7c10 2023-10-30T12:17:57.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.761+0000 7fb6feffd640 1 --2- 172.21.15.148:0/429013307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7000b7c10 0x7fb7000ae850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c730 tx=0x7fb6f000cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:57.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.761+0000 7fb6fcff9640 10 client.?.objecter ms_handle_connect 0x7fb7000b7c10 2023-10-30T12:17:57.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.761+0000 7fb6fcff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:57.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 1 -- 172.21.15.148:0/429013307 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f00058b0 con 0x7fb7000b7c10 2023-10-30T12:17:57.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:57.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:57.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 10 monclient: dump: 2023-10-30T12:17:57.781 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:57.782 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:57.782 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.782 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.782 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:57.782 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:57.783 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:57.783 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:57.783 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:57.783 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:57.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 10 monclient: _finish_auth 0 2023-10-30T12:17:57.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 10 monclient: _check_auth_tickets 2023-10-30T12:17:57.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:57.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 1 -- 172.21.15.148:0/429013307 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f000e040 con 0x7fb7000b7c10 2023-10-30T12:17:57.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb7058e0640 5 monclient: authenticate success, global_id 4270 2023-10-30T12:17:57.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:57.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb7058e0640 10 monclient: _renew_subs 2023-10-30T12:17:57.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 1 -- 172.21.15.148:0/429013307 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0005ca0 con 0x7fb7000b7c10 2023-10-30T12:17:57.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb7058e0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb7058e0640 1 -- 172.21.15.148:0/429013307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7000ac9e0 con 0x7fb7000b7c10 2023-10-30T12:17:57.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:57.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:57.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fcff9640 10 monclient: dump: 2023-10-30T12:17:57.786 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:57.786 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:57.786 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.786 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:57.787 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:57.787 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:57.787 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:57.787 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:57.787 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:57.788 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:57.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.764+0000 7fb6fdffb640 4 set_mon_vals no callback set 2023-10-30T12:17:57.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.765+0000 7fb7058e0640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:57.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.765+0000 7fb7058e0640 10 monclient: _renew_subs 2023-10-30T12:17:57.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.765+0000 7fb7058e0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.765+0000 7fb7058e0640 1 -- 172.21.15.148:0/429013307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7000ab330 con 0x7fb7000b7c10 2023-10-30T12:17:57.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.765+0000 7fb7058e0640 1 librados: init done 2023-10-30T12:17:57.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.765+0000 7fb7058e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:57.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.765+0000 7fb7058e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:57.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.765+0000 7fb7058e0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:57.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.765+0000 7fb7058e0640 1 -- 172.21.15.148:0/429013307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7000ae130 con 0x7fb7000b7c10 2023-10-30T12:17:57.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.783+0000 7fb6fcff9640 1 -- 172.21.15.148:0/429013307 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fb6f0007070 con 0x7fb7000b7c10 2023-10-30T12:17:57.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.784+0000 7fb6fcff9640 1 --2- 172.21.15.148:0/429013307 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb6e006c7b0 0x7fb6e006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:57.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.784+0000 7fb6f7fff640 1 --2- 172.21.15.148:0/429013307 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb6e006c7b0 0x7fb6e006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:57.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.784+0000 7fb6f7fff640 10 monclient: get_auth_request con 0x7fb6e006c7b0 auth_method 0 2023-10-30T12:17:57.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.784+0000 7fb6f7fff640 1 --2- 172.21.15.148:0/429013307 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb6e006c7b0 0x7fb6e006ec70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb70013aa80 tx=0x7fb6e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:57.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.784+0000 7fb6fcff9640 10 client.4270.objecter ms_handle_connect 0x7fb6e006c7b0 2023-10-30T12:17:57.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.785+0000 7fb6fcff9640 1 -- 172.21.15.148:0/429013307 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3737+0+0 (secure 0 0 0) 0x7fb6f0057740 con 0x7fb7000b7c10 2023-10-30T12:17:57.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.785+0000 7fb6fcff9640 10 client.4270.objecter ms_dispatch 0x7fb700000b90 osd_map(20..20 src has 1..20) v4 2023-10-30T12:17:57.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.785+0000 7fb6fcff9640 3 client.4270.objecter handle_osd_map got epochs [20,20] > 0 2023-10-30T12:17:57.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.785+0000 7fb6fcff9640 3 client.4270.objecter handle_osd_map decoding full epoch 20 2023-10-30T12:17:57.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:57.785+0000 7fb6fcff9640 20 client.4270.objecter dump_active .. 0 homeless 2023-10-30T12:17:58.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:58.036+0000 7fb6fcff9640 1 -- 172.21.15.148:0/429013307 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb6f005b7e0 con 0x7fb7000b7c10 2023-10-30T12:17:58.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:58.036+0000 7fb6fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:58.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:58.036+0000 7fb6fcff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:58.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:58.231+0000 7fb7058e0640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T12:17:58.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:58.231+0000 7fb7058e0640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T12:17:58.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:58.231+0000 7fb7058e0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:17:58.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:58.231+0000 7fb7058e0640 1 -- 172.21.15.148:0/429013307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fb7000afa10 con 0x7fb7000b7c10 2023-10-30T12:17:59.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.147+0000 7fb6fcff9640 1 -- 172.21.15.148:0/429013307 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v21) v1 ==== 109+0+0 (secure 0 0 0) 0x7fb6f005f990 con 0x7fb7000b7c10 2023-10-30T12:17:59.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.147+0000 7fb6fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T12:17:59.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.147+0000 7fb6fcff9640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-10-30T12:17:59.149 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'rbd' created 2023-10-30T12:17:59.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.149+0000 7fb7058e0640 10 librados: watch_flush enter 2023-10-30T12:17:59.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.149+0000 7fb7058e0640 10 librados: watch_flush exit 2023-10-30T12:17:59.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.149+0000 7fb7058e0640 20 client.4270.objecter shutdown clearing up homeless session... 2023-10-30T12:17:59.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 10 client.4270.objecter successfully canceled tick 2023-10-30T12:17:59.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 -- 172.21.15.148:0/429013307 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb6e006c7b0 msgr2=0x7fb6e006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:59.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 --2- 172.21.15.148:0/429013307 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb6e006c7b0 0x7fb6e006ec70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb70013aa80 tx=0x7fb6e8009040 comp rx=0 tx=0).stop 2023-10-30T12:17:59.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 10 monclient: shutdown 2023-10-30T12:17:59.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:59.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 -- 172.21.15.148:0/429013307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7000b7c10 msgr2=0x7fb7000ae850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:59.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 --2- 172.21.15.148:0/429013307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7000b7c10 0x7fb7000ae850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c730 tx=0x7fb6f000cbf0 comp rx=0 tx=0).stop 2023-10-30T12:17:59.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 -- 172.21.15.148:0/429013307 shutdown_connections 2023-10-30T12:17:59.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 --2- 172.21.15.148:0/429013307 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb6e006c7b0 0x7fb6e006ec70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:59.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 --2- 172.21.15.148:0/429013307 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb70014a460 0x7fb7000aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:59.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 --2- 172.21.15.148:0/429013307 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb700141940 0x7fb7000ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:59.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 --2- 172.21.15.148:0/429013307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7000b7c10 0x7fb7000ae850 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:59.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 -- 172.21.15.148:0/429013307 >> 172.21.15.148:0/429013307 conn(0x7fb70006c660 msgr2=0x7fb70013b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:59.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 -- 172.21.15.148:0/429013307 shutdown_connections 2023-10-30T12:17:59.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 -- 172.21.15.148:0/429013307 wait complete. 2023-10-30T12:17:59.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.150+0000 7fb7058e0640 1 librados: shutdown 2023-10-30T12:17:59.163 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-30T12:17:59.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.268+0000 7f77b8ba9640 10 monclient: get_monmap_and_config 2023-10-30T12:17:59.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.268+0000 7f77b8ba9640 10 monclient: build_initial_monmap 2023-10-30T12:17:59.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.268+0000 7f77b8ba9640 10 monclient: monmap: 2023-10-30T12:17:59.271 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:17:59.271 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:59.271 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:17:59.269907+0000 2023-10-30T12:17:59.271 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:17:59.269907+0000 2023-10-30T12:17:59.271 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:59.272 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:59.272 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:17:59.272 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:17:59.272 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:17:59.272 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:17:59.272 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:17:59.273 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:59.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.269+0000 7f77b8ba9640 1 Processor -- start 2023-10-30T12:17:59.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.269+0000 7f77b8ba9640 1 -- start start 2023-10-30T12:17:59.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.269+0000 7f77b8ba9640 10 monclient: init 2023-10-30T12:17:59.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:59.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T12:17:59.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f77b40b7c10 0x7f77b40a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:59.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 10 monclient(hunting): picked mon.noname-c con 0x7f77b40b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:17:59.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77b4141940 addr v1:172.21.15.148:6789/0 2023-10-30T12:17:59.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f77b413e730 0x7f77b413eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:59.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 10 monclient(hunting): picked mon.noname-a con 0x7f77b413e730 addr v2:172.21.15.148:3300/0 2023-10-30T12:17:59.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b40a9350 con 0x7f77b4141940 2023-10-30T12:17:59.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:59.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b40670a0 con 0x7f77b413e730 2023-10-30T12:17:59.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:59.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b40b7ff0 con 0x7f77b40b7c10 2023-10-30T12:17:59.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:59.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b8ba9640 10 monclient(hunting): authenticate will time out at 1571.475156s 2023-10-30T12:17:59.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77abfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f77b4141940 0x7f77b4141d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59000/0 (socket says 172.21.15.148:59000) 2023-10-30T12:17:59.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77abfff640 1 -- 172.21.15.148:0/3449149656 learned_addr learned my addr 172.21.15.148:0/3449149656 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:17:59.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.270+0000 7f77b259c640 1 --2- 172.21.15.148:0/3449149656 >> v2:172.21.15.150:3300/0 conn(0x7f77b40b7c10 0x7f77b40a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:59.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.271+0000 7f77b259c640 10 monclient(hunting): get_auth_request con 0x7f77b40b7c10 auth_method 0 2023-10-30T12:17:59.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.271+0000 7f77b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:59.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.271+0000 7f77b259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:59.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.271+0000 7f77b259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:59.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.271+0000 7f77b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:59.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.271+0000 7f77b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:59.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.271+0000 7f77b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:59.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b2d9d640 1 --2- 172.21.15.148:0/3449149656 >> v2:172.21.15.148:3300/0 conn(0x7f77b413e730 0x7f77b413eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:59.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b1d9b640 1 -- 172.21.15.148:0/3449149656 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341949413 0 0) 0x7f779c001110 con 0x7f77b4141940 2023-10-30T12:17:59.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b1d9b640 10 monclient(hunting): _negotiate 2023-10-30T12:17:59.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:59.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:59.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b1d9b640 10 monclient(hunting): my global_id is 4260 2023-10-30T12:17:59.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b1d9b640 1 -- 172.21.15.148:0/3449149656 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a0003520 con 0x7f77b4141940 2023-10-30T12:17:59.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b259c640 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2023-10-30T12:17:59.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b259c640 1 -- 172.21.15.148:0/3449149656 >> v2:172.21.15.148:3300/0 conn(0x7f77b413e730 msgr2=0x7f77b413eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:59.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b259c640 1 --2- 172.21.15.148:0/3449149656 >> v2:172.21.15.148:3300/0 conn(0x7f77b413e730 0x7f77b413eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:59.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b259c640 1 -- 172.21.15.148:0/3449149656 >> v1:172.21.15.148:6789/0 conn(0x7f77b4141940 legacy=0x7f77b4141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:59.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b259c640 10 monclient: _finish_hunting 0 2023-10-30T12:17:59.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b259c640 1 monclient: found mon.noname-c 2023-10-30T12:17:59.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:59.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:17:59.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.274+0000 7f77b259c640 1 -- 172.21.15.148:0/3449149656 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b40aa6a0 con 0x7f77b40b7c10 2023-10-30T12:17:59.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.276+0000 7f77b259c640 1 --2- 172.21.15.148:0/3449149656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b40b7c10 0x7f77b40a8aa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f77b40b4d80 tx=0x7f77a402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db2da39135e82ddf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:59.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 1 -- 172.21.15.148:0/3449149656 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a40315a0 con 0x7f77b40b7c10 2023-10-30T12:17:59.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:59.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:59.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 10 monclient: dump: 2023-10-30T12:17:59.286 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:59.288 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:59.288 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:59.288 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:59.288 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:59.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 10 monclient: _finish_auth 0 2023-10-30T12:17:59.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:17:59.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:59.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 1 -- 172.21.15.148:0/3449149656 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a4031710 con 0x7f77b40b7c10 2023-10-30T12:17:59.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b8ba9640 5 monclient: authenticate success, global_id 4279 2023-10-30T12:17:59.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b8ba9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:59.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:59.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b8ba9640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:59.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 1 -- 172.21.15.148:0/3449149656 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4031ab0 con 0x7f77b40b7c10 2023-10-30T12:17:59.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b8ba9640 4 set_mon_vals no callback set 2023-10-30T12:17:59.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:59.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.283+0000 7f77b1d9b640 10 monclient: dump: 2023-10-30T12:17:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:59.292 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:59.292 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:59.292 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:59.292 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:59.292 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:59.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.284+0000 7f77b8ba9640 10 monclient: shutdown 2023-10-30T12:17:59.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.284+0000 7f77b8ba9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:59.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.284+0000 7f77b8ba9640 1 -- 172.21.15.148:0/3449149656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b40b7c10 msgr2=0x7f77b40a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:59.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.284+0000 7f77b8ba9640 1 --2- 172.21.15.148:0/3449149656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b40b7c10 0x7f77b40a8aa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f77b40b4d80 tx=0x7f77a402b840 comp rx=0 tx=0).stop 2023-10-30T12:17:59.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.284+0000 7f77b8ba9640 1 -- 172.21.15.148:0/3449149656 shutdown_connections 2023-10-30T12:17:59.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.284+0000 7f77b8ba9640 1 --2- 172.21.15.148:0/3449149656 >> v2:172.21.15.148:3300/0 conn(0x7f77b413e730 0x7f77b413eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:59.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.284+0000 7f77b8ba9640 1 --2- 172.21.15.148:0/3449149656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b40b7c10 0x7f77b40a8aa0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:59.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.284+0000 7f77b8ba9640 1 -- 172.21.15.148:0/3449149656 >> 172.21.15.148:0/3449149656 conn(0x7f77b406c660 msgr2=0x7f77b40b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:59.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.284+0000 7f77b8ba9640 1 -- 172.21.15.148:0/3449149656 shutdown_connections 2023-10-30T12:17:59.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.284+0000 7f77b8ba9640 1 -- 172.21.15.148:0/3449149656 wait complete. 2023-10-30T12:17:59.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.285+0000 7f77b8ba9640 10 monclient: build_initial_monmap 2023-10-30T12:17:59.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.285+0000 7f77b8ba9640 10 monclient: monmap: 2023-10-30T12:17:59.299 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:17:59.299 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:59.299 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:17:59.299 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:17:59.299 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:59.300 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:17:59.300 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:17:59.300 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:17:59.300 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:17:59.300 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:59.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.285+0000 7f77b8ba9640 1 librados: starting msgr at 2023-10-30T12:17:59.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.285+0000 7f77b8ba9640 1 librados: starting objecter 2023-10-30T12:17:59.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.285+0000 7f77b8ba9640 1 Processor -- start 2023-10-30T12:17:59.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.285+0000 7f77b8ba9640 1 -- start start 2023-10-30T12:17:59.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.285+0000 7f77b8ba9640 1 librados: setting wanted keys 2023-10-30T12:17:59.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.285+0000 7f77b8ba9640 1 librados: calling monclient init 2023-10-30T12:17:59.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.285+0000 7f77b8ba9640 10 monclient: init 2023-10-30T12:17:59.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:59.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:17:59.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b40b7c10 0x7f77b40afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:59.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 10 monclient(hunting): picked mon.noname-b con 0x7f77b40b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:17:59.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77b413e730 0x7f77b40aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:59.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 10 monclient(hunting): picked mon.noname-a con 0x7f77b413e730 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:17:59.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77b4141940 0x7f77b41d6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:59.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 10 monclient(hunting): picked mon.noname-c con 0x7f77b4141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:17:59.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:59.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b40a87d0 con 0x7f77b413e730 2023-10-30T12:17:59.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:59.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b40adf20 con 0x7f77b40b7c10 2023-10-30T12:17:59.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:59.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b41d6920 con 0x7f77b4141940 2023-10-30T12:17:59.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:59.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.286+0000 7f77b8ba9640 10 monclient(hunting): authenticate will time out at 1571.491526s 2023-10-30T12:17:59.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b40b7c10 0x7f77b40afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:59.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b2d9d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77b4141940 0x7f77b41d6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:59.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b40b7c10 0x7f77b40afae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:57788/0 (socket says 172.21.15.148:57788) 2023-10-30T12:17:59.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b259c640 1 -- 172.21.15.148:0/1973286510 learned_addr learned my addr 172.21.15.148:0/1973286510 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:17:59.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b259c640 10 monclient(hunting): get_auth_request con 0x7f77b40b7c10 auth_method 0 2023-10-30T12:17:59.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:59.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:59.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:59.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b2d9d640 10 monclient(hunting): get_auth_request con 0x7f77b4141940 auth_method 0 2023-10-30T12:17:59.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:59.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:59.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.287+0000 7f77b2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:59.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.288+0000 7f77b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:59.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.288+0000 7f77b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:59.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.288+0000 7f77b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:59.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.288+0000 7f77b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:59.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.288+0000 7f77b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:59.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.288+0000 7f77b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:59.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.289+0000 7f77abfff640 1 --2- 172.21.15.148:0/1973286510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77b413e730 0x7f77b40aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:59.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.289+0000 7f77abfff640 10 monclient(hunting): get_auth_request con 0x7f77b413e730 auth_method 0 2023-10-30T12:17:59.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.289+0000 7f77abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:59.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.289+0000 7f77abfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:59.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.289+0000 7f77abfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:59.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.290+0000 7f77abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:59.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.290+0000 7f77abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:59.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.290+0000 7f77abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:59.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.291+0000 7f77b2d9d640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2023-10-30T12:17:59.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.291+0000 7f77b2d9d640 1 -- 172.21.15.148:0/1973286510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b40b7c10 msgr2=0x7f77b40afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:59.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.291+0000 7f77b2d9d640 1 --2- 172.21.15.148:0/1973286510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b40b7c10 0x7f77b40afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:59.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.291+0000 7f77b2d9d640 1 -- 172.21.15.148:0/1973286510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77b413e730 msgr2=0x7f77b40aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:59.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.291+0000 7f77b2d9d640 1 --2- 172.21.15.148:0/1973286510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77b413e730 0x7f77b40aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:59.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.291+0000 7f77b2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:17:59.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.291+0000 7f77b2d9d640 1 monclient: found mon.noname-c 2023-10-30T12:17:59.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.291+0000 7f77b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:59.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.291+0000 7f77b2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:17:59.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.291+0000 7f77b2d9d640 1 -- 172.21.15.148:0/1973286510 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b40ac120 con 0x7f77b4141940 2023-10-30T12:17:59.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.293+0000 7f77b2d9d640 1 --2- 172.21.15.148:0/1973286510 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77b4141940 0x7f77b41d6e40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00ccb0 tx=0x7f77ac00a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:59.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.293+0000 7f77aaffd640 10 client.?.objecter ms_handle_connect 0x7f77b4141940 2023-10-30T12:17:59.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.293+0000 7f77aaffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:59.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 1 -- 172.21.15.148:0/1973286510 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac003b90 con 0x7f77b4141940 2023-10-30T12:17:59.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:59.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:59.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 10 monclient: dump: 2023-10-30T12:17:59.315 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:59.315 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:59.316 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:59.316 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:59.316 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:59.316 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:59.316 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:59.317 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:59.317 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:59.317 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:59.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 10 monclient: _finish_auth 0 2023-10-30T12:17:59.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:59.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:59.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 1 -- 172.21.15.148:0/1973286510 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ac003d00 con 0x7f77b4141940 2023-10-30T12:17:59.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:59.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 1 -- 172.21.15.148:0/1973286510 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac004240 con 0x7f77b4141940 2023-10-30T12:17:59.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:59.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:59.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77aaffd640 10 monclient: dump: 2023-10-30T12:17:59.319 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:17:59.319 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:17:59.319 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:59.319 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:17:59.319 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:59.320 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:17:59.320 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:17:59.320 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:17:59.320 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:17:59.320 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:17:59.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77b8ba9640 5 monclient: authenticate success, global_id 4226 2023-10-30T12:17:59.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77b8ba9640 10 monclient: _renew_subs 2023-10-30T12:17:59.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77b8ba9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:59.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77b159a640 4 set_mon_vals no callback set 2023-10-30T12:17:59.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77b8ba9640 1 -- 172.21.15.148:0/1973286510 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b40af2a0 con 0x7f77b4141940 2023-10-30T12:17:59.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77b8ba9640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:59.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77b8ba9640 10 monclient: _renew_subs 2023-10-30T12:17:59.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77b8ba9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:59.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77b8ba9640 1 -- 172.21.15.148:0/1973286510 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b41d7560 con 0x7f77b4141940 2023-10-30T12:17:59.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.297+0000 7f77b8ba9640 1 librados: init done 2023-10-30T12:17:59.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.298+0000 7f77b8ba9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:59.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.298+0000 7f77b8ba9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:59.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.298+0000 7f77b8ba9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:59.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.298+0000 7f77b8ba9640 1 -- 172.21.15.148:0/1973286510 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b40a97c0 con 0x7f77b4141940 2023-10-30T12:17:59.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.320+0000 7f77aaffd640 1 -- 172.21.15.148:0/1973286510 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7f77ac0043b0 con 0x7f77b4141940 2023-10-30T12:17:59.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.320+0000 7f77aaffd640 1 --2- 172.21.15.148:0/1973286510 >> v2:172.21.15.148:6800/3205090179 conn(0x7f779406c780 0x7f779406ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:59.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.321+0000 7f77b259c640 1 --2- 172.21.15.148:0/1973286510 >> v2:172.21.15.148:6800/3205090179 conn(0x7f779406c780 0x7f779406ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:59.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.321+0000 7f77b259c640 10 monclient: get_auth_request con 0x7f779406c780 auth_method 0 2023-10-30T12:17:59.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.321+0000 7f77b259c640 1 --2- 172.21.15.148:0/1973286510 >> v2:172.21.15.148:6800/3205090179 conn(0x7f779406c780 0x7f779406ec40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009560 tx=0x7f77a403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:59.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.321+0000 7f77aaffd640 10 client.4226.objecter ms_handle_connect 0x7f779406c780 2023-10-30T12:17:59.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.325+0000 7f77aaffd640 1 -- 172.21.15.148:0/1973286510 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4079+0+0 (secure 0 0 0) 0x7f77ac024080 con 0x7f77b4141940 2023-10-30T12:17:59.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.325+0000 7f77aaffd640 10 client.4226.objecter ms_dispatch 0x7f77b4000b90 osd_map(21..21 src has 1..21) v4 2023-10-30T12:17:59.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.325+0000 7f77aaffd640 3 client.4226.objecter handle_osd_map got epochs [21,21] > 0 2023-10-30T12:17:59.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.325+0000 7f77aaffd640 3 client.4226.objecter handle_osd_map decoding full epoch 21 2023-10-30T12:17:59.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.325+0000 7f77aaffd640 20 client.4226.objecter dump_active .. 0 homeless 2023-10-30T12:17:59.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.582+0000 7f77aaffd640 1 -- 172.21.15.148:0/1973286510 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f77ac058a60 con 0x7f77b4141940 2023-10-30T12:17:59.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.582+0000 7f77aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:59.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.582+0000 7f77aaffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.786+0000 7f77b8ba9640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T12:17:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.786+0000 7f77b8ba9640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T12:17:59.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.786+0000 7f77b8ba9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:17:59.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:17:59.786+0000 7f77b8ba9640 1 -- 172.21.15.148:0/1973286510 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f77b40a97c0 con 0x7f77b4141940 2023-10-30T12:17:59.965 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:01:15.039 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:00.024 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:01:15.098 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:00.046 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:01:15.093 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:00.049 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:01:15.123 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:00.097 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:01:15.144 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:00.104 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:01:15.178 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:00.281 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:01:15.354 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:00.309 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:01:15.356 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:00.328 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:01:15.402 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:00.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.739+0000 7f77aaffd640 1 -- 172.21.15.148:0/1973286510 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v22) v1 ==== 173+0+0 (secure 0 0 0) 0x7f77ac05cb00 con 0x7f77b4141940 2023-10-30T12:18:00.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.739+0000 7f77aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T12:18:00.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.739+0000 7f77aaffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-10-30T12:18:00.741 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-30T12:18:00.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 10 librados: watch_flush enter 2023-10-30T12:18:00.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 10 librados: watch_flush exit 2023-10-30T12:18:00.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 20 client.4226.objecter shutdown clearing up homeless session... 2023-10-30T12:18:00.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 10 client.4226.objecter successfully canceled tick 2023-10-30T12:18:00.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 -- 172.21.15.148:0/1973286510 >> v2:172.21.15.148:6800/3205090179 conn(0x7f779406c780 msgr2=0x7f779406ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:00.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 --2- 172.21.15.148:0/1973286510 >> v2:172.21.15.148:6800/3205090179 conn(0x7f779406c780 0x7f779406ec40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009560 tx=0x7f77a403d040 comp rx=0 tx=0).stop 2023-10-30T12:18:00.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 10 monclient: shutdown 2023-10-30T12:18:00.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:00.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 -- 172.21.15.148:0/1973286510 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77b4141940 msgr2=0x7f77b41d6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:00.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 --2- 172.21.15.148:0/1973286510 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77b4141940 0x7f77b41d6e40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00ccb0 tx=0x7f77ac00a660 comp rx=0 tx=0).stop 2023-10-30T12:18:00.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 -- 172.21.15.148:0/1973286510 shutdown_connections 2023-10-30T12:18:00.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 --2- 172.21.15.148:0/1973286510 >> v2:172.21.15.148:6800/3205090179 conn(0x7f779406c780 0x7f779406ec40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:00.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 --2- 172.21.15.148:0/1973286510 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77b4141940 0x7f77b41d6e40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:00.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 --2- 172.21.15.148:0/1973286510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77b413e730 0x7f77b40aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:00.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 --2- 172.21.15.148:0/1973286510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b40b7c10 0x7f77b40afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:00.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 -- 172.21.15.148:0/1973286510 >> 172.21.15.148:0/1973286510 conn(0x7f77b406c660 msgr2=0x7f77b40b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:00.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 -- 172.21.15.148:0/1973286510 shutdown_connections 2023-10-30T12:18:00.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 -- 172.21.15.148:0/1973286510 wait complete. 2023-10-30T12:18:00.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.742+0000 7f77b8ba9640 1 librados: shutdown 2023-10-30T12:18:00.754 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T12:18:00.755 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T12:18:00.755 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T12:18:00.756 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T12:18:00.756 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T12:18:00.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.862+0000 7fd1a5c61640 10 monclient: get_monmap_and_config 2023-10-30T12:18:00.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.862+0000 7fd1a5c61640 10 monclient: build_initial_monmap 2023-10-30T12:18:00.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.862+0000 7fd1a5c61640 10 monclient: monmap: 2023-10-30T12:18:00.864 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:00.864 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:00.864 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:00.863179+0000 2023-10-30T12:18:00.865 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:00.863179+0000 2023-10-30T12:18:00.865 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:00.865 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:00.865 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:00.865 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:00.866 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:00.866 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:00.866 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:00.866 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.862+0000 7fd1a5c61640 1 Processor -- start 2023-10-30T12:18:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.862+0000 7fd1a5c61640 1 -- start start 2023-10-30T12:18:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.862+0000 7fd1a5c61640 10 monclient: init 2023-10-30T12:18:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:00.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-10-30T12:18:00.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1a00b7c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:00.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a0141940 0x7fd1a0141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:00.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a0141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:00.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd1a013e730 0x7fd1a013eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:00.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a013e730 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:00.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a00a9350 con 0x7fd1a00b7c10 2023-10-30T12:18:00.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:00.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a00670a0 con 0x7fd1a013e730 2023-10-30T12:18:00.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:00.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a00b7ff0 con 0x7fd1a0141940 2023-10-30T12:18:00.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:00.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.863+0000 7fd1a5c61640 10 monclient(hunting): authenticate will time out at 1573.068448s 2023-10-30T12:18:00.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19effd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a0141940 0x7fd1a0141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:00.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd1a00b7c10 0x7fd1a00a8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:48188/0 (socket says 172.21.15.148:48188) 2023-10-30T12:18:00.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19f7fe640 1 -- 172.21.15.148:0/1241245823 learned_addr learned my addr 172.21.15.148:0/1241245823 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:18:00.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19ffff640 1 --2- 172.21.15.148:0/1241245823 >> v2:172.21.15.148:3300/0 conn(0x7fd1a013e730 0x7fd1a013eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:00.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19effd640 10 monclient(hunting): get_auth_request con 0x7fd1a0141940 auth_method 0 2023-10-30T12:18:00.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:00.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:00.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:00.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19ffff640 10 monclient(hunting): get_auth_request con 0x7fd1a013e730 auth_method 0 2023-10-30T12:18:00.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:00.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:00.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.864+0000 7fd19ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:00.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.865+0000 7fd19ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:00.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.865+0000 7fd19ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:00.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.865+0000 7fd19ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:00.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.865+0000 7fd19effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:00.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.865+0000 7fd19effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:00.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.865+0000 7fd19effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:00.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.866+0000 7fd19e7fc640 1 -- 172.21.15.148:0/1241245823 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049651590 0 0) 0x7fd190001110 con 0x7fd1a00b7c10 2023-10-30T12:18:00.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.866+0000 7fd19e7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:18:00.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.866+0000 7fd19e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:00.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.866+0000 7fd19e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:00.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.866+0000 7fd19e7fc640 10 monclient(hunting): my global_id is 4285 2023-10-30T12:18:00.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.866+0000 7fd19e7fc640 1 -- 172.21.15.148:0/1241245823 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd188003520 con 0x7fd1a00b7c10 2023-10-30T12:18:00.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.867+0000 7fd19effd640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2023-10-30T12:18:00.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.867+0000 7fd19effd640 1 -- 172.21.15.148:0/1241245823 >> v2:172.21.15.148:3300/0 conn(0x7fd1a013e730 msgr2=0x7fd1a013eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:00.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.867+0000 7fd19effd640 1 --2- 172.21.15.148:0/1241245823 >> v2:172.21.15.148:3300/0 conn(0x7fd1a013e730 0x7fd1a013eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:00.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.867+0000 7fd19effd640 1 -- 172.21.15.148:0/1241245823 >> v1:172.21.15.150:6789/0 conn(0x7fd1a00b7c10 legacy=0x7fd1a00a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:00.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.867+0000 7fd19effd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:00.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.867+0000 7fd19effd640 1 monclient: found mon.noname-b 2023-10-30T12:18:00.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.867+0000 7fd19effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:00.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.867+0000 7fd19effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:00.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.867+0000 7fd19effd640 1 -- 172.21.15.148:0/1241245823 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a00aa6a0 con 0x7fd1a0141940 2023-10-30T12:18:00.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.869+0000 7fd19effd640 1 --2- 172.21.15.148:0/1241245823 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a0141940 0x7fd1a0141d60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd1940097a0 tx=0x7fd19402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e4dad772239c2c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:00.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 1 -- 172.21.15.148:0/1241245823 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd194031780 con 0x7fd1a0141940 2023-10-30T12:18:00.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:00.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:00.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 10 monclient: dump: 2023-10-30T12:18:00.885 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:00.885 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:00.885 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:00.885 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:00.887 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:00.887 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:00.887 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:00.888 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:00.888 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:00.888 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:00.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:00.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:00.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:00.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 1 -- 172.21.15.148:0/1241245823 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1940318f0 con 0x7fd1a0141940 2023-10-30T12:18:00.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd1a5c61640 5 monclient: authenticate success, global_id 4229 2023-10-30T12:18:00.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd1a5c61640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:00.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:00.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd1a5c61640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:00.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 1 -- 172.21.15.148:0/1241245823 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd194031c90 con 0x7fd1a0141940 2023-10-30T12:18:00.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd1a5c61640 4 set_mon_vals no callback set 2023-10-30T12:18:00.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:00.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:00.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd19e7fc640 10 monclient: dump: 2023-10-30T12:18:00.891 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:00.891 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:00.891 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:00.891 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:00.891 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:00.892 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:00.892 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:00.892 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:00.892 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:00.892 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:00.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd1a5c61640 10 monclient: shutdown 2023-10-30T12:18:00.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd1a5c61640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:00.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.873+0000 7fd1a5c61640 1 -- 172.21.15.148:0/1241245823 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a0141940 msgr2=0x7fd1a0141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:00.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.874+0000 7fd1a5c61640 1 --2- 172.21.15.148:0/1241245823 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a0141940 0x7fd1a0141d60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd1940097a0 tx=0x7fd19402b820 comp rx=0 tx=0).stop 2023-10-30T12:18:00.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.874+0000 7fd1a5c61640 1 -- 172.21.15.148:0/1241245823 shutdown_connections 2023-10-30T12:18:00.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.874+0000 7fd1a5c61640 1 --2- 172.21.15.148:0/1241245823 >> v2:172.21.15.148:3300/0 conn(0x7fd1a013e730 0x7fd1a013eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:00.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.874+0000 7fd1a5c61640 1 --2- 172.21.15.148:0/1241245823 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a0141940 0x7fd1a0141d60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:00.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.874+0000 7fd1a5c61640 1 -- 172.21.15.148:0/1241245823 >> 172.21.15.148:0/1241245823 conn(0x7fd1a006c660 msgr2=0x7fd1a00b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:00.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.874+0000 7fd1a5c61640 1 -- 172.21.15.148:0/1241245823 shutdown_connections 2023-10-30T12:18:00.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.874+0000 7fd1a5c61640 1 -- 172.21.15.148:0/1241245823 wait complete. 2023-10-30T12:18:00.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.874+0000 7fd1a5c61640 10 monclient: build_initial_monmap 2023-10-30T12:18:00.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.874+0000 7fd1a5c61640 10 monclient: monmap: 2023-10-30T12:18:00.895 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:00.895 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:00.895 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:00.895 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:00.895 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:00.896 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:00.896 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:00.896 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:00.896 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:00.896 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:00.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 librados: starting msgr at 2023-10-30T12:18:00.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 librados: starting objecter 2023-10-30T12:18:00.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 Processor -- start 2023-10-30T12:18:00.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 -- start start 2023-10-30T12:18:00.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 librados: setting wanted keys 2023-10-30T12:18:00.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 librados: calling monclient init 2023-10-30T12:18:00.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient: init 2023-10-30T12:18:00.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:00.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:00.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1a00b7c10 0x7fd1a00b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:00.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a00b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:00.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a013e730 0x7fd1a011f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:00.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a013e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:00.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd1a0141940 0x7fd1a01d6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:00.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a0141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:00.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:00.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a013f4e0 con 0x7fd1a0141940 2023-10-30T12:18:00.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:00.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a01d68b0 con 0x7fd1a00b7c10 2023-10-30T12:18:00.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:00.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a011f8d0 con 0x7fd1a013e730 2023-10-30T12:18:00.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:00.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.875+0000 7fd1a5c61640 10 monclient(hunting): authenticate will time out at 1573.080880s 2023-10-30T12:18:00.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1a00b7c10 0x7fd1a00b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:00.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd1a0141940 0x7fd1a01d6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:00.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19effd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a013e730 0x7fd1a011f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:00.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1a00b7c10 0x7fd1a00b0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48586/0 (socket says 172.21.15.148:48586) 2023-10-30T12:18:00.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19f7fe640 1 -- 172.21.15.148:0/307369968 learned_addr learned my addr 172.21.15.148:0/307369968 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:00.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19f7fe640 10 monclient(hunting): get_auth_request con 0x7fd1a00b7c10 auth_method 0 2023-10-30T12:18:00.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:00.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:00.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:00.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19ffff640 10 monclient(hunting): get_auth_request con 0x7fd1a0141940 auth_method 0 2023-10-30T12:18:00.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:00.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:00.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.876+0000 7fd19ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:00.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19effd640 10 monclient(hunting): get_auth_request con 0x7fd1a013e730 auth_method 0 2023-10-30T12:18:00.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:00.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:00.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:00.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:00.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:00.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:00.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:00.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:00.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:00.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:00.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:00.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.877+0000 7fd19effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:00.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.879+0000 7fd19f7fe640 10 monclient(hunting): handle_auth_done global_id 4288 payload 824 2023-10-30T12:18:00.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.879+0000 7fd19f7fe640 1 -- 172.21.15.148:0/307369968 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a013e730 msgr2=0x7fd1a011f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:00.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.879+0000 7fd19f7fe640 1 --2- 172.21.15.148:0/307369968 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a013e730 0x7fd1a011f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:00.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.879+0000 7fd19f7fe640 1 -- 172.21.15.148:0/307369968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd1a0141940 msgr2=0x7fd1a01d6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:00.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.879+0000 7fd19f7fe640 1 --2- 172.21.15.148:0/307369968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd1a0141940 0x7fd1a01d6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:00.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.879+0000 7fd19f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:18:00.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.879+0000 7fd19f7fe640 1 monclient: found mon.noname-b 2023-10-30T12:18:00.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.879+0000 7fd19f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:00.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.879+0000 7fd19f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:00.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.879+0000 7fd19f7fe640 1 -- 172.21.15.148:0/307369968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a00adfa0 con 0x7fd1a00b7c10 2023-10-30T12:18:00.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.882+0000 7fd19f7fe640 1 --2- 172.21.15.148:0/307369968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1a00b7c10 0x7fd1a00b0d20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd19000c760 tx=0x7fd19000cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:00.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.882+0000 7fd187fff640 10 client.?.objecter ms_handle_connect 0x7fd1a00b7c10 2023-10-30T12:18:00.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.882+0000 7fd187fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:00.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 1 -- 172.21.15.148:0/307369968 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd190005ab0 con 0x7fd1a00b7c10 2023-10-30T12:18:00.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:00.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:00.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 10 monclient: dump: 2023-10-30T12:18:00.910 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:00.911 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:00.911 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:00.911 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:00.911 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:00.911 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:00.911 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:00.912 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:00.912 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:00.912 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:00.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 10 monclient: _finish_auth 0 2023-10-30T12:18:00.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:00.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:00.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 1 -- 172.21.15.148:0/307369968 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19000e040 con 0x7fd1a00b7c10 2023-10-30T12:18:00.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:00.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 1 -- 172.21.15.148:0/307369968 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd190005e20 con 0x7fd1a00b7c10 2023-10-30T12:18:00.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:00.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:00.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd187fff640 10 monclient: dump: 2023-10-30T12:18:00.914 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:00.914 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:00.914 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:00.914 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:00.915 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:00.915 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:00.915 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:00.915 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:00.915 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:00.916 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:00.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd1a5c61640 5 monclient: authenticate success, global_id 4288 2023-10-30T12:18:00.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd1a5c61640 10 monclient: _renew_subs 2023-10-30T12:18:00.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd1a5c61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:00.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd1a5c61640 1 -- 172.21.15.148:0/307369968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a0120d10 con 0x7fd1a00b7c10 2023-10-30T12:18:00.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd19dffb640 4 set_mon_vals no callback set 2023-10-30T12:18:00.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd1a5c61640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:00.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd1a5c61640 10 monclient: _renew_subs 2023-10-30T12:18:00.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd1a5c61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:00.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd1a5c61640 1 -- 172.21.15.148:0/307369968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a01d7420 con 0x7fd1a00b7c10 2023-10-30T12:18:00.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.885+0000 7fd1a5c61640 1 librados: init done 2023-10-30T12:18:00.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.886+0000 7fd1a5c61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:00.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.886+0000 7fd1a5c61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:00.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.886+0000 7fd1a5c61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:00.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.886+0000 7fd1a5c61640 1 -- 172.21.15.148:0/307369968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a0075550 con 0x7fd1a00b7c10 2023-10-30T12:18:00.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.906+0000 7fd187fff640 1 -- 172.21.15.148:0/307369968 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fd190007070 con 0x7fd1a00b7c10 2023-10-30T12:18:00.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.906+0000 7fd187fff640 1 --2- 172.21.15.148:0/307369968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd18c06c780 0x7fd18c06ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:00.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.906+0000 7fd19effd640 1 --2- 172.21.15.148:0/307369968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd18c06c780 0x7fd18c06ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:00.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.906+0000 7fd19effd640 10 monclient: get_auth_request con 0x7fd18c06c780 auth_method 0 2023-10-30T12:18:00.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.907+0000 7fd19effd640 1 --2- 172.21.15.148:0/307369968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd18c06c780 0x7fd18c06ec40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd194036000 tx=0x7fd19403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:00.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.907+0000 7fd187fff640 10 client.4288.objecter ms_handle_connect 0x7fd18c06c780 2023-10-30T12:18:00.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.909+0000 7fd187fff640 1 -- 172.21.15.148:0/307369968 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fd190057790 con 0x7fd1a00b7c10 2023-10-30T12:18:00.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.909+0000 7fd187fff640 10 client.4288.objecter ms_dispatch 0x7fd1a0000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:18:00.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.909+0000 7fd187fff640 3 client.4288.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:18:00.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.909+0000 7fd187fff640 3 client.4288.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:18:00.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:00.909+0000 7fd187fff640 20 client.4288.objecter dump_active .. 0 homeless 2023-10-30T12:18:01.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.162+0000 7fd187fff640 1 -- 172.21.15.148:0/307369968 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd19005b830 con 0x7fd1a00b7c10 2023-10-30T12:18:01.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.162+0000 7fd187fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:01.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.162+0000 7fd187fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:01.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.472+0000 7fd1a5c61640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-10-30T12:18:01.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.473+0000 7fd1a5c61640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-10-30T12:18:01.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.473+0000 7fd1a5c61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:01.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.473+0000 7fd1a5c61640 1 -- 172.21.15.148:0/307369968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fd1a01d7420 con 0x7fd1a00b7c10 2023-10-30T12:18:01.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.627+0000 7fd187fff640 1 -- 172.21.15.148:0/307369968 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+170882 (secure 0 0 0) 0x7fd19005f9e0 con 0x7fd1a00b7c10 2023-10-30T12:18:01.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.627+0000 7fd187fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-10-30T12:18:01.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.627+0000 7fd187fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:01.632 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:18:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 10 librados: watch_flush enter 2023-10-30T12:18:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 10 librados: watch_flush exit 2023-10-30T12:18:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 20 client.4288.objecter shutdown clearing up homeless session... 2023-10-30T12:18:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 10 client.4288.objecter successfully canceled tick 2023-10-30T12:18:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 -- 172.21.15.148:0/307369968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd18c06c780 msgr2=0x7fd18c06ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 --2- 172.21.15.148:0/307369968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd18c06c780 0x7fd18c06ec40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd194036000 tx=0x7fd19403d040 comp rx=0 tx=0).stop 2023-10-30T12:18:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 10 monclient: shutdown 2023-10-30T12:18:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 -- 172.21.15.148:0/307369968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1a00b7c10 msgr2=0x7fd1a00b0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 --2- 172.21.15.148:0/307369968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1a00b7c10 0x7fd1a00b0d20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd19000c760 tx=0x7fd19000cc20 comp rx=0 tx=0).stop 2023-10-30T12:18:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 -- 172.21.15.148:0/307369968 shutdown_connections 2023-10-30T12:18:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 --2- 172.21.15.148:0/307369968 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd18c06c780 0x7fd18c06ec40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 --2- 172.21.15.148:0/307369968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd1a0141940 0x7fd1a01d6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 --2- 172.21.15.148:0/307369968 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd1a013e730 0x7fd1a011f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 --2- 172.21.15.148:0/307369968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1a00b7c10 0x7fd1a00b0d20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 -- 172.21.15.148:0/307369968 >> 172.21.15.148:0/307369968 conn(0x7fd1a006c660 msgr2=0x7fd1a013db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 -- 172.21.15.148:0/307369968 shutdown_connections 2023-10-30T12:18:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 -- 172.21.15.148:0/307369968 wait complete. 2023-10-30T12:18:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.630+0000 7fd1a5c61640 1 librados: shutdown 2023-10-30T12:18:01.644 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":4,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":3205090179}]},"active_addr":"172.21.15.148:6800/3205090179","active_change":"2023-10-30T12:17:08.814393+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4115,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.148:0","nonce":2714580555}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.148:0","nonce":4054402651}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.148:0","nonce":4036069473}]}]} 2023-10-30T12:18:01.650 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T12:18:01.650 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T12:18:01.650 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:18:01.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.763+0000 7f4e46ca2640 10 monclient: get_monmap_and_config 2023-10-30T12:18:01.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.763+0000 7f4e46ca2640 10 monclient: build_initial_monmap 2023-10-30T12:18:01.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.763+0000 7f4e46ca2640 10 monclient: monmap: 2023-10-30T12:18:01.765 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:01.765 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:01.766 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:01.764581+0000 2023-10-30T12:18:01.766 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:01.764581+0000 2023-10-30T12:18:01.766 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:01.766 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:01.766 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:01.767 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:01.767 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:01.767 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:01.767 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:01.767 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:01.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 1 Processor -- start 2023-10-30T12:18:01.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 1 -- start start 2023-10-30T12:18:01.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 10 monclient: init 2023-10-30T12:18:01.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:01.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T12:18:01.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4e400b7c10 0x7f4e400a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:01.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 10 monclient(hunting): picked mon.noname-b con 0x7f4e400b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e40141940 0x7f4e40141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 10 monclient(hunting): picked mon.noname-c con 0x7f4e40141940 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e400b1c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e400a9350 con 0x7f4e400b1c10 2023-10-30T12:18:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e400670a0 con 0x7f4e40141940 2023-10-30T12:18:01.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:01.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400b7ff0 con 0x7f4e400b7c10 2023-10-30T12:18:01.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:01.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.764+0000 7f4e46ca2640 10 monclient(hunting): authenticate will time out at 1573.969863s 2023-10-30T12:18:01.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.765+0000 7f4e3ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e40141940 0x7f4e40141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:01.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.766+0000 7f4e3ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e40141940 0x7f4e40141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48602/0 (socket says 172.21.15.148:48602) 2023-10-30T12:18:01.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.766+0000 7f4e3ffff640 1 -- 172.21.15.148:0/2290060215 learned_addr learned my addr 172.21.15.148:0/2290060215 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:01.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.766+0000 7f4e3ffff640 10 monclient(hunting): get_auth_request con 0x7f4e40141940 auth_method 0 2023-10-30T12:18:01.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.766+0000 7f4e3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:01.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.766+0000 7f4e3ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:01.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.766+0000 7f4e3ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:01.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.767+0000 7f4e3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:01.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.767+0000 7f4e3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:01.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.767+0000 7f4e3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:01.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.769+0000 7f4e3ffff640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2023-10-30T12:18:01.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.769+0000 7f4e3ffff640 1 -- 172.21.15.148:0/2290060215 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4e400b7c10 msgr2=0x7f4e400a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:01.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.769+0000 7f4e3ffff640 1 --2- 172.21.15.148:0/2290060215 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4e400b7c10 0x7f4e400a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:01.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.769+0000 7f4e3ffff640 1 -- 172.21.15.148:0/2290060215 >> v1:172.21.15.150:6789/0 conn(0x7f4e400b1c10 legacy=0x7f4e400b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:01.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.769+0000 7f4e3ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:01.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.769+0000 7f4e3ffff640 1 monclient: found mon.noname-c 2023-10-30T12:18:01.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.769+0000 7f4e3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:01.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.769+0000 7f4e3ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:01.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.769+0000 7f4e3ffff640 1 -- 172.21.15.148:0/2290060215 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e400aa6a0 con 0x7f4e40141940 2023-10-30T12:18:01.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.772+0000 7f4e3ffff640 1 --2- 172.21.15.148:0/2290060215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e40141940 0x7f4e40141d60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009870 tx=0x7f4e2c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a31fde79f61171c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:01.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 1 -- 172.21.15.148:0/2290060215 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c031640 con 0x7f4e40141940 2023-10-30T12:18:01.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:01.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:01.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 10 monclient: dump: 2023-10-30T12:18:01.779 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:01.779 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:01.779 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:01.779 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:01.779 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:01.779 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:01.780 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:01.780 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:01.780 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:01.780 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:01.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:18:01.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:18:01.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:01.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 1 -- 172.21.15.148:0/2290060215 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c0317b0 con 0x7f4e40141940 2023-10-30T12:18:01.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e46ca2640 5 monclient: authenticate success, global_id 4291 2023-10-30T12:18:01.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e46ca2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:01.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:01.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 1 -- 172.21.15.148:0/2290060215 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c031b50 con 0x7f4e40141940 2023-10-30T12:18:01.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:01.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:01.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e3f7fe640 10 monclient: dump: 2023-10-30T12:18:01.782 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:01.783 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:01.783 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:01.783 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:01.783 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:01.783 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:01.784 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:01.784 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:01.784 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:01.784 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:01.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e46ca2640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:01.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.776+0000 7f4e46ca2640 4 set_mon_vals no callback set 2023-10-30T12:18:01.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.777+0000 7f4e46ca2640 10 monclient: shutdown 2023-10-30T12:18:01.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.777+0000 7f4e46ca2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:01.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.777+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2290060215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e40141940 msgr2=0x7f4e40141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:01.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.777+0000 7f4e46ca2640 1 --2- 172.21.15.148:0/2290060215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e40141940 0x7f4e40141d60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009870 tx=0x7f4e2c02b8f0 comp rx=0 tx=0).stop 2023-10-30T12:18:01.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.777+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2290060215 shutdown_connections 2023-10-30T12:18:01.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.777+0000 7f4e46ca2640 1 --2- 172.21.15.148:0/2290060215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e40141940 0x7f4e40141d60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:01.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.777+0000 7f4e46ca2640 1 --2- 172.21.15.148:0/2290060215 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4e400b7c10 0x7f4e400a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:01.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.777+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2290060215 >> 172.21.15.148:0/2290060215 conn(0x7f4e4006c660 msgr2=0x7f4e400b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:01.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.777+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2290060215 shutdown_connections 2023-10-30T12:18:01.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.777+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2290060215 wait complete. 2023-10-30T12:18:01.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.778+0000 7f4e46ca2640 10 monclient: build_initial_monmap 2023-10-30T12:18:01.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.778+0000 7f4e46ca2640 10 monclient: monmap: 2023-10-30T12:18:01.790 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:01.790 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:01.790 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:01.790 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:01.790 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:01.791 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:01.791 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:01.791 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:01.791 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:01.791 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:01.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.778+0000 7f4e46ca2640 1 librados: starting msgr at 2023-10-30T12:18:01.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.778+0000 7f4e46ca2640 1 librados: starting objecter 2023-10-30T12:18:01.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.779+0000 7f4e46ca2640 1 Processor -- start 2023-10-30T12:18:01.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.779+0000 7f4e46ca2640 1 -- start start 2023-10-30T12:18:01.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.779+0000 7f4e46ca2640 1 librados: setting wanted keys 2023-10-30T12:18:01.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.779+0000 7f4e46ca2640 1 librados: calling monclient init 2023-10-30T12:18:01.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.779+0000 7f4e46ca2640 10 monclient: init 2023-10-30T12:18:01.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:01.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:18:01.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4e400b1c10 0x7f4e401c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:01.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 10 monclient(hunting): picked mon.noname-c con 0x7f4e400b1c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:01.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e400b7c10 0x7f4e4013a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:01.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 10 monclient(hunting): picked mon.noname-b con 0x7f4e400b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:01.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e40141940 0x7f4e4013a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:01.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 10 monclient(hunting): picked mon.noname-a con 0x7f4e40141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:01.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:01.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400b1880 con 0x7f4e40141940 2023-10-30T12:18:01.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:01.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e401c9730 con 0x7f4e400b7c10 2023-10-30T12:18:01.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:01.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400b1160 con 0x7f4e400b1c10 2023-10-30T12:18:01.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:01.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e46ca2640 10 monclient(hunting): authenticate will time out at 1573.985091s 2023-10-30T12:18:01.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.780+0000 7f4e4523f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e40141940 0x7f4e4013a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:01.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e3ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e400b7c10 0x7f4e4013a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:01.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e4523f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e40141940 0x7f4e4013a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59768/0 (socket says 172.21.15.148:59768) 2023-10-30T12:18:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e4523f640 1 -- 172.21.15.148:0/2769058875 learned_addr learned my addr 172.21.15.148:0/2769058875 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e4523f640 10 monclient(hunting): get_auth_request con 0x7f4e40141940 auth_method 0 2023-10-30T12:18:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e4523f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e4523f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e4523f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e3ffff640 10 monclient(hunting): get_auth_request con 0x7f4e400b7c10 auth_method 0 2023-10-30T12:18:01.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:01.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e3ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:01.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e3ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:01.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e4523f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:01.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e4523f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:01.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e4523f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:01.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:01.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:01.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.781+0000 7f4e3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:01.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.784+0000 7f4e3ffff640 10 monclient(hunting): handle_auth_done global_id 4294 payload 824 2023-10-30T12:18:01.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.784+0000 7f4e3ffff640 1 -- 172.21.15.148:0/2769058875 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4e400b1c10 msgr2=0x7f4e401c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:01.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.784+0000 7f4e3ffff640 1 --2- 172.21.15.148:0/2769058875 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4e400b1c10 0x7f4e401c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:01.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.784+0000 7f4e3ffff640 1 -- 172.21.15.148:0/2769058875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e40141940 msgr2=0x7f4e4013a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:01.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.784+0000 7f4e3ffff640 1 --2- 172.21.15.148:0/2769058875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e40141940 0x7f4e4013a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:01.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.784+0000 7f4e3ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:01.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.784+0000 7f4e3ffff640 1 monclient: found mon.noname-b 2023-10-30T12:18:01.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.784+0000 7f4e3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:01.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.784+0000 7f4e3ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:01.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.784+0000 7f4e3ffff640 1 -- 172.21.15.148:0/2769058875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e401c99a0 con 0x7f4e400b7c10 2023-10-30T12:18:01.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.786+0000 7f4e3ffff640 1 --2- 172.21.15.148:0/2769058875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e400b7c10 0x7f4e4013a440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c033000 tx=0x7f4e2c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:01.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.786+0000 7f4e3d7fa640 10 client.?.objecter ms_handle_connect 0x7f4e400b7c10 2023-10-30T12:18:01.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.787+0000 7f4e3d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:01.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 1 -- 172.21.15.148:0/2769058875 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c03ac60 con 0x7f4e400b7c10 2023-10-30T12:18:01.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:01.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:01.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 10 monclient: dump: 2023-10-30T12:18:01.804 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:01.804 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:01.804 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:01.805 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:01.805 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:01.805 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:01.805 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:01.806 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:01.806 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:01.806 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:01.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:18:01.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:18:01.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:01.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 1 -- 172.21.15.148:0/2769058875 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c03e030 con 0x7f4e400b7c10 2023-10-30T12:18:01.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:01.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 1 -- 172.21.15.148:0/2769058875 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0443f0 con 0x7f4e400b7c10 2023-10-30T12:18:01.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:01.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:01.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3d7fa640 10 monclient: dump: 2023-10-30T12:18:01.808 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:01.808 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:01.808 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:01.808 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:01.809 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:01.809 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:01.809 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:01.809 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:01.809 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:01.809 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:01.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e46ca2640 5 monclient: authenticate success, global_id 4294 2023-10-30T12:18:01.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e46ca2640 10 monclient: _renew_subs 2023-10-30T12:18:01.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e3effd640 4 set_mon_vals no callback set 2023-10-30T12:18:01.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e46ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:01.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2769058875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4013aec0 con 0x7f4e400b7c10 2023-10-30T12:18:01.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e46ca2640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:01.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e46ca2640 10 monclient: _renew_subs 2023-10-30T12:18:01.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e46ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:01.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2769058875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e400b0bb0 con 0x7f4e400b7c10 2023-10-30T12:18:01.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.790+0000 7f4e46ca2640 1 librados: init done 2023-10-30T12:18:01.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.791+0000 7f4e46ca2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:01.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.791+0000 7f4e46ca2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:01.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.791+0000 7f4e46ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:01.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:01.791+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2769058875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e40075550 con 0x7f4e400b7c10 2023-10-30T12:18:02.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.038+0000 7f4e3d7fa640 1 -- 172.21.15.148:0/2769058875 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7f4e2c02f050 con 0x7f4e400b7c10 2023-10-30T12:18:02.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.038+0000 7f4e3d7fa640 1 --2- 172.21.15.148:0/2769058875 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4e2406c780 0x7f4e2406ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:02.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.038+0000 7f4e44a3e640 1 --2- 172.21.15.148:0/2769058875 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4e2406c780 0x7f4e2406ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:02.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.038+0000 7f4e44a3e640 10 monclient: get_auth_request con 0x7f4e2406c780 auth_method 0 2023-10-30T12:18:02.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.038+0000 7f4e44a3e640 1 --2- 172.21.15.148:0/2769058875 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4e2406c780 0x7f4e2406ec40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009710 tx=0x7f4e34006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:02.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.038+0000 7f4e3d7fa640 10 client.4294.objecter ms_handle_connect 0x7f4e2406c780 2023-10-30T12:18:02.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.040+0000 7f4e3d7fa640 1 -- 172.21.15.148:0/2769058875 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f4e2c052080 con 0x7f4e400b7c10 2023-10-30T12:18:02.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.040+0000 7f4e3d7fa640 10 client.4294.objecter ms_dispatch 0x7f4e40000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:18:02.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.040+0000 7f4e3d7fa640 3 client.4294.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:18:02.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.040+0000 7f4e3d7fa640 3 client.4294.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:18:02.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.040+0000 7f4e3d7fa640 20 client.4294.objecter dump_active .. 0 homeless 2023-10-30T12:18:02.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.051+0000 7f4e3d7fa640 1 -- 172.21.15.148:0/2769058875 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4e2c07a350 con 0x7f4e400b7c10 2023-10-30T12:18:02.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.051+0000 7f4e3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:02.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.051+0000 7f4e3d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:02.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.234+0000 7f4e46ca2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:18:02.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.234+0000 7f4e46ca2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:18:02.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.234+0000 7f4e46ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:02.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.234+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2769058875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e400aa6a0 con 0x7f4e400b7c10 2023-10-30T12:18:02.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.346+0000 7f4e3d7fa640 1 -- 172.21.15.148:0/2769058875 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f4e2c075a40 con 0x7f4e400b7c10 2023-10-30T12:18:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.384+0000 7f4e3d7fa640 1 -- 172.21.15.148:0/2769058875 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+12775 (secure 0 0 0) 0x7f4e2c009490 con 0x7f4e400b7c10 2023-10-30T12:18:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.384+0000 7f4e3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:18:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.384+0000 7f4e3d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:02.386 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:18:02.386 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":23,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:18:01.598154+0000","last_up_change":"2023-10-30T12:17:51.437771+0000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:17:53.315604+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T12:17:58.245447+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":2395694124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2395694124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2395694124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2395694124}]},"public_addr":"172.21.15.148:6805/2395694124","cluster_addr":"172.21.15.148:6807/2395694124","heartbeat_back_addr":"172.21.15.148:6810/2395694124","heartbeat_front_addr":"172.21.15.148:6808/2395694124","state":["exists","up"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":912412250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":912412250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":912412250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":912412250}]},"public_addr":"172.21.15.148:6813/912412250","cluster_addr":"172.21.15.148:6814/912412250","heartbeat_back_addr":"172.21.15.148:6816/912412250","heartbeat_front_addr":"172.21.15.148:6815/912412250","state":["exists","up"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":2368806934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2368806934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":2368806934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":2368806934}]},"public_addr":"172.21.15.148:6801/2368806934","cluster_addr":"172.21.15.148:6802/2368806934","heartbeat_back_addr":"172.21.15.148:6804/2368806934","heartbeat_front_addr":"172.21.15.148:6803/2368806934","state":["exists","up"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":2233349848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":2233349848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2233349848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":2233349848}]},"public_addr":"172.21.15.148:6806/2233349848","cluster_addr":"172.21.15.148:6809/2233349848","heartbeat_back_addr":"172.21.15.148:6812/2233349848","heartbeat_front_addr":"172.21.15.148:6811/2233349848","state":["exists","up"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2836839739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":2836839739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":2836839739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2836839739}]},"public_addr":"172.21.15.150:6808/2836839739","cluster_addr":"172.21.15.150:6809/2836839739","heartbeat_back_addr":"172.21.15.150:6811/2836839739","heartbeat_front_addr":"172.21.15.150:6810/2836839739","state":["exists","up"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4190376848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":4190376848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6807","nonce":4190376848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4190376848}]},"public_addr":"172.21.15.150:6804/4190376848","cluster_addr":"172.21.15.150:6805/4190376848","heartbeat_back_addr":"172.21.15.150:6807/4190376848","heartbeat_front_addr":"172.21.15.150:6806/4190376848","state":["exists","up"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2604225934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6801","nonce":2604225934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":2604225934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2604225934}]},"public_addr":"172.21.15.150:6800/2604225934","cluster_addr":"172.21.15.150:6801/2604225934","heartbeat_back_addr":"172.21.15.150:6803/2604225934","heartbeat_front_addr":"172.21.15.150:6802/2604225934","state":["exists","up"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3463178370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":3463178370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":3463178370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3463178370}]},"public_addr":"172.21.15.150:6812/3463178370","cluster_addr":"172.21.15.150:6813/3463178370","heartbeat_back_addr":"172.21.15.150:6815/3463178370","heartbeat_front_addr":"172.21.15.150:6814/3463178370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.920964+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.821114+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.365893+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:49.037497+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.184019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.644370+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:47.688638+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.429422+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:18:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.385+0000 7f4e46ca2640 10 librados: watch_flush enter 2023-10-30T12:18:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.385+0000 7f4e46ca2640 10 librados: watch_flush exit 2023-10-30T12:18:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.385+0000 7f4e46ca2640 20 client.4294.objecter shutdown clearing up homeless session... 2023-10-30T12:18:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.385+0000 7f4e46ca2640 10 client.4294.objecter successfully canceled tick 2023-10-30T12:18:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.385+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2769058875 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4e2406c780 msgr2=0x7f4e2406ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.385+0000 7f4e46ca2640 1 --2- 172.21.15.148:0/2769058875 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4e2406c780 0x7f4e2406ec40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009710 tx=0x7f4e34006c90 comp rx=0 tx=0).stop 2023-10-30T12:18:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.385+0000 7f4e46ca2640 10 monclient: shutdown 2023-10-30T12:18:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.385+0000 7f4e46ca2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.385+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2769058875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e400b7c10 msgr2=0x7f4e4013a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.385+0000 7f4e46ca2640 1 --2- 172.21.15.148:0/2769058875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e400b7c10 0x7f4e4013a440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c033000 tx=0x7f4e2c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.386+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2769058875 shutdown_connections 2023-10-30T12:18:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.386+0000 7f4e46ca2640 1 --2- 172.21.15.148:0/2769058875 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4e2406c780 0x7f4e2406ec40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.386+0000 7f4e46ca2640 1 --2- 172.21.15.148:0/2769058875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e40141940 0x7f4e4013a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.386+0000 7f4e46ca2640 1 --2- 172.21.15.148:0/2769058875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e400b7c10 0x7f4e4013a440 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.386+0000 7f4e46ca2640 1 --2- 172.21.15.148:0/2769058875 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4e400b1c10 0x7f4e401c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.386+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2769058875 >> 172.21.15.148:0/2769058875 conn(0x7f4e4006c660 msgr2=0x7f4e4013f840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.386+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2769058875 shutdown_connections 2023-10-30T12:18:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.386+0000 7f4e46ca2640 1 -- 172.21.15.148:0/2769058875 wait complete. 2023-10-30T12:18:02.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.386+0000 7f4e46ca2640 1 librados: shutdown 2023-10-30T12:18:02.398 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T12:18:02.399 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:18:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.498+0000 7fab9047c640 10 monclient: get_monmap_and_config 2023-10-30T12:18:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.498+0000 7fab9047c640 10 monclient: build_initial_monmap 2023-10-30T12:18:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.498+0000 7fab9047c640 10 monclient: monmap: 2023-10-30T12:18:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:02.499131+0000 2023-10-30T12:18:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:02.499131+0000 2023-10-30T12:18:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:02.503 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.498+0000 7fab9047c640 1 Processor -- start 2023-10-30T12:18:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.498+0000 7fab9047c640 1 -- start start 2023-10-30T12:18:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.498+0000 7fab9047c640 10 monclient: init 2023-10-30T12:18:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T12:18:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 0x7fab880a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 10 monclient(hunting): picked mon.noname-b con 0x7fab880b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab88141940 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab8813e730 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab880a9350 con 0x7fab88141940 2023-10-30T12:18:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab880aa6a0 con 0x7fab8813e730 2023-10-30T12:18:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab880670a0 con 0x7fab880b7c10 2023-10-30T12:18:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.499+0000 7fab9047c640 10 monclient(hunting): authenticate will time out at 1574.704827s 2023-10-30T12:18:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.500+0000 7fab8da17640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fab88141940 0x7fab88141d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52928/0 (socket says 172.21.15.148:52928) 2023-10-30T12:18:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.500+0000 7fab8da17640 1 -- 172.21.15.148:0/3749825861 learned_addr learned my addr 172.21.15.148:0/3749825861 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:18:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.500+0000 7fab8e218640 1 --2- 172.21.15.148:0/3749825861 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 0x7fab880a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.501+0000 7fab8e218640 10 monclient(hunting): get_auth_request con 0x7fab880b7c10 auth_method 0 2023-10-30T12:18:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.501+0000 7fab8e218640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:02.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.501+0000 7fab8e218640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:02.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.501+0000 7fab8e218640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:02.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.501+0000 7fab8e218640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.501+0000 7fab8e218640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.501+0000 7fab8e218640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8d216640 1 -- 172.21.15.148:0/3749825861 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051407511 0 0) 0x7fab78001110 con 0x7fab88141940 2023-10-30T12:18:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8d216640 10 monclient(hunting): _negotiate 2023-10-30T12:18:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8d216640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8d216640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8d216640 10 monclient(hunting): my global_id is 4275 2023-10-30T12:18:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8d216640 1 -- 172.21.15.148:0/3749825861 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab74003520 con 0x7fab88141940 2023-10-30T12:18:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8e218640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2023-10-30T12:18:02.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8e218640 1 -- 172.21.15.148:0/3749825861 >> v1:172.21.15.150:6789/0 conn(0x7fab8813e730 legacy=0x7fab8813eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:02.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8e218640 1 -- 172.21.15.148:0/3749825861 >> v1:172.21.15.148:6789/0 conn(0x7fab88141940 legacy=0x7fab88141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:02.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8e218640 10 monclient: _finish_hunting 0 2023-10-30T12:18:02.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8e218640 1 monclient: found mon.noname-b 2023-10-30T12:18:02.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8e218640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:02.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8e218640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:02.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.503+0000 7fab8e218640 1 -- 172.21.15.148:0/3749825861 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab880aa910 con 0x7fab880b7c10 2023-10-30T12:18:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.506+0000 7fab8e218640 1 --2- 172.21.15.148:0/3749825861 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 0x7fab880a8aa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab840097d0 tx=0x7fab8402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3ddbcbf35b1f7ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.508+0000 7fab8d216640 1 -- 172.21.15.148:0/3749825861 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84031800 con 0x7fab880b7c10 2023-10-30T12:18:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.508+0000 7fab8d216640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.508+0000 7fab8d216640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.508+0000 7fab8d216640 10 monclient: dump: 2023-10-30T12:18:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:02.525 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.508+0000 7fab8d216640 10 monclient: _finish_auth 0 2023-10-30T12:18:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.508+0000 7fab8d216640 10 monclient: _check_auth_tickets 2023-10-30T12:18:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.508+0000 7fab8d216640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.508+0000 7fab9047c640 5 monclient: authenticate success, global_id 4235 2023-10-30T12:18:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.508+0000 7fab9047c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab8d216640 1 -- 172.21.15.148:0/3749825861 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab84031970 con 0x7fab880b7c10 2023-10-30T12:18:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab8d216640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab8d216640 1 -- 172.21.15.148:0/3749825861 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84031cf0 con 0x7fab880b7c10 2023-10-30T12:18:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab9047c640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab9047c640 4 set_mon_vals no callback set 2023-10-30T12:18:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab8d216640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab8d216640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab8d216640 10 monclient: dump: 2023-10-30T12:18:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:02.529 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab9047c640 10 monclient: shutdown 2023-10-30T12:18:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab9047c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab9047c640 1 -- 172.21.15.148:0/3749825861 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 msgr2=0x7fab880a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab9047c640 1 --2- 172.21.15.148:0/3749825861 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 0x7fab880a8aa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab840097d0 tx=0x7fab8402b8a0 comp rx=0 tx=0).stop 2023-10-30T12:18:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab9047c640 1 -- 172.21.15.148:0/3749825861 shutdown_connections 2023-10-30T12:18:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab9047c640 1 --2- 172.21.15.148:0/3749825861 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 0x7fab880a8aa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.510+0000 7fab9047c640 1 -- 172.21.15.148:0/3749825861 >> 172.21.15.148:0/3749825861 conn(0x7fab8806c660 msgr2=0x7fab880b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.511+0000 7fab9047c640 1 -- 172.21.15.148:0/3749825861 shutdown_connections 2023-10-30T12:18:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.511+0000 7fab9047c640 1 -- 172.21.15.148:0/3749825861 wait complete. 2023-10-30T12:18:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.511+0000 7fab9047c640 10 monclient: build_initial_monmap 2023-10-30T12:18:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.511+0000 7fab9047c640 10 monclient: monmap: 2023-10-30T12:18:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:02.533 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.511+0000 7fab9047c640 1 librados: starting msgr at 2023-10-30T12:18:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.511+0000 7fab9047c640 1 librados: starting objecter 2023-10-30T12:18:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.511+0000 7fab9047c640 1 Processor -- start 2023-10-30T12:18:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 1 -- start start 2023-10-30T12:18:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 1 librados: setting wanted keys 2023-10-30T12:18:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 1 librados: calling monclient init 2023-10-30T12:18:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient: init 2023-10-30T12:18:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 0x7fab881c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:02.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient(hunting): picked mon.noname-c con 0x7fab880b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:02.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab8813e730 0x7fab881c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:02.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient(hunting): picked mon.noname-a con 0x7fab8813e730 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:02.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab88141940 0x7fab881d6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:02.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient(hunting): picked mon.noname-b con 0x7fab88141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:02.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:02.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8813c2a0 con 0x7fab8813e730 2023-10-30T12:18:02.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:02.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab881c8220 con 0x7fab88141940 2023-10-30T12:18:02.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab880aff80 con 0x7fab880b7c10 2023-10-30T12:18:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.512+0000 7fab9047c640 10 monclient(hunting): authenticate will time out at 1574.717465s 2023-10-30T12:18:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8e218640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 0x7fab881c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8da17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab8813e730 0x7fab881c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8ea19640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab88141940 0x7fab881d6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8e218640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 0x7fab881c8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54384/0 (socket says 172.21.15.148:54384) 2023-10-30T12:18:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8e218640 1 -- 172.21.15.148:0/1164135969 learned_addr learned my addr 172.21.15.148:0/1164135969 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:02.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8e218640 10 monclient(hunting): get_auth_request con 0x7fab880b7c10 auth_method 0 2023-10-30T12:18:02.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8e218640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:02.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8e218640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:02.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8e218640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:02.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8da17640 10 monclient(hunting): get_auth_request con 0x7fab8813e730 auth_method 0 2023-10-30T12:18:02.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8da17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:02.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8da17640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:02.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8da17640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:02.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8ea19640 10 monclient(hunting): get_auth_request con 0x7fab88141940 auth_method 0 2023-10-30T12:18:02.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8ea19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:02.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8ea19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:02.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8ea19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:02.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8e218640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:02.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8e218640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:02.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8e218640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:02.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8da17640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:02.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8da17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:02.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8da17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:02.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8ea19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:02.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8ea19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:02.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.513+0000 7fab8ea19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:02.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.516+0000 7fab8da17640 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2023-10-30T12:18:02.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.516+0000 7fab8da17640 1 -- 172.21.15.148:0/1164135969 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 msgr2=0x7fab881c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:02.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.516+0000 7fab8da17640 1 --2- 172.21.15.148:0/1164135969 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 0x7fab881c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:02.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.516+0000 7fab8da17640 1 -- 172.21.15.148:0/1164135969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab88141940 msgr2=0x7fab881d6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:02.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.516+0000 7fab8da17640 1 --2- 172.21.15.148:0/1164135969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab88141940 0x7fab881d6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:02.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.516+0000 7fab8da17640 10 monclient: _finish_hunting 0 2023-10-30T12:18:02.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.516+0000 7fab8da17640 1 monclient: found mon.noname-a 2023-10-30T12:18:02.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.516+0000 7fab8da17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:02.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.516+0000 7fab8da17640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:02.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.516+0000 7fab8da17640 1 -- 172.21.15.148:0/1164135969 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8813b120 con 0x7fab8813e730 2023-10-30T12:18:02.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.519+0000 7fab8da17640 1 --2- 172.21.15.148:0/1164135969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab8813e730 0x7fab881c7ce0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fab7800c3c0 tx=0x7fab7800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:02.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.519+0000 7fab7effd640 10 client.?.objecter ms_handle_connect 0x7fab8813e730 2023-10-30T12:18:02.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.519+0000 7fab7effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:02.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 1 -- 172.21.15.148:0/1164135969 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78005ce0 con 0x7fab8813e730 2023-10-30T12:18:02.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:02.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:02.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 10 monclient: dump: 2023-10-30T12:18:02.551 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:02.551 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:02.551 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:02.551 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:02.552 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:02.552 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:02.552 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:02.552 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:02.552 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:02.553 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:02.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 10 monclient: _finish_auth 0 2023-10-30T12:18:02.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:02.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:02.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 1 -- 172.21.15.148:0/1164135969 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7800e040 con 0x7fab8813e730 2023-10-30T12:18:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 1 -- 172.21.15.148:0/1164135969 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab780072f0 con 0x7fab8813e730 2023-10-30T12:18:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab9047c640 5 monclient: authenticate success, global_id 4278 2023-10-30T12:18:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab9047c640 10 monclient: _renew_subs 2023-10-30T12:18:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab9047c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:02.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab9047c640 1 -- 172.21.15.148:0/1164135969 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab881d7320 con 0x7fab8813e730 2023-10-30T12:18:02.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab8ca15640 4 set_mon_vals no callback set 2023-10-30T12:18:02.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:02.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:02.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab7effd640 10 monclient: dump: 2023-10-30T12:18:02.556 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:02.556 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:02.556 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:02.556 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:02.556 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:02.557 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:02.557 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:02.557 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:02.557 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:02.557 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:02.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab9047c640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:02.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab9047c640 10 monclient: _renew_subs 2023-10-30T12:18:02.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab9047c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:02.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab9047c640 1 -- 172.21.15.148:0/1164135969 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab880b0da0 con 0x7fab8813e730 2023-10-30T12:18:02.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.523+0000 7fab9047c640 1 librados: init done 2023-10-30T12:18:02.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.524+0000 7fab9047c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:02.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.524+0000 7fab9047c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:02.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.524+0000 7fab9047c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:02.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.524+0000 7fab9047c640 1 -- 172.21.15.148:0/1164135969 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab88075550 con 0x7fab8813e730 2023-10-30T12:18:02.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.542+0000 7fab7effd640 1 -- 172.21.15.148:0/1164135969 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fab78007460 con 0x7fab8813e730 2023-10-30T12:18:02.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.543+0000 7fab7effd640 1 --2- 172.21.15.148:0/1164135969 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6c06c810 0x7fab6c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:02.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.543+0000 7fab8e218640 1 --2- 172.21.15.148:0/1164135969 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6c06c810 0x7fab6c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:02.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.543+0000 7fab8e218640 10 monclient: get_auth_request con 0x7fab6c06c810 auth_method 0 2023-10-30T12:18:02.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.543+0000 7fab8e218640 1 --2- 172.21.15.148:0/1164135969 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6c06c810 0x7fab6c06ecd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fab84000c00 tx=0x7fab8403e040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:02.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.543+0000 7fab7effd640 10 client.4278.objecter ms_handle_connect 0x7fab6c06c810 2023-10-30T12:18:02.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.779+0000 7fab7effd640 1 -- 172.21.15.148:0/1164135969 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fab780577c0 con 0x7fab8813e730 2023-10-30T12:18:02.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.779+0000 7fab7effd640 10 client.4278.objecter ms_dispatch 0x7fab88000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T12:18:02.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.779+0000 7fab7effd640 3 client.4278.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T12:18:02.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.779+0000 7fab7effd640 3 client.4278.objecter handle_osd_map decoding full epoch 23 2023-10-30T12:18:02.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.779+0000 7fab7effd640 20 client.4278.objecter dump_active .. 0 homeless 2023-10-30T12:18:02.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.779+0000 7fab7effd640 1 -- 172.21.15.148:0/1164135969 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab7805b860 con 0x7fab8813e730 2023-10-30T12:18:02.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.779+0000 7fab7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:02.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.779+0000 7fab7effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:02.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.974+0000 7fab9047c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:18:02.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.974+0000 7fab9047c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:18:02.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.974+0000 7fab9047c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:02.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:02.974+0000 7fab9047c640 1 -- 172.21.15.148:0/1164135969 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab881d7320 con 0x7fab8813e730 2023-10-30T12:18:03.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.068+0000 7fab7effd640 1 -- 172.21.15.148:0/1164135969 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+12762 (secure 0 0 0) 0x7fab7805fa10 con 0x7fab8813e730 2023-10-30T12:18:03.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.068+0000 7fab7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:18:03.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.068+0000 7fab7effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:03.071 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:18:03.071 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":24,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:18:02.808394+0000","last_up_change":"2023-10-30T12:17:51.437771+0000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:17:53.315604+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T12:17:58.245447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":2395694124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2395694124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2395694124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2395694124}]},"public_addr":"172.21.15.148:6805/2395694124","cluster_addr":"172.21.15.148:6807/2395694124","heartbeat_back_addr":"172.21.15.148:6810/2395694124","heartbeat_front_addr":"172.21.15.148:6808/2395694124","state":["exists","up"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":912412250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":912412250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":912412250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":912412250}]},"public_addr":"172.21.15.148:6813/912412250","cluster_addr":"172.21.15.148:6814/912412250","heartbeat_back_addr":"172.21.15.148:6816/912412250","heartbeat_front_addr":"172.21.15.148:6815/912412250","state":["exists","up"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":2368806934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2368806934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":2368806934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":2368806934}]},"public_addr":"172.21.15.148:6801/2368806934","cluster_addr":"172.21.15.148:6802/2368806934","heartbeat_back_addr":"172.21.15.148:6804/2368806934","heartbeat_front_addr":"172.21.15.148:6803/2368806934","state":["exists","up"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":2233349848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":2233349848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2233349848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":2233349848}]},"public_addr":"172.21.15.148:6806/2233349848","cluster_addr":"172.21.15.148:6809/2233349848","heartbeat_back_addr":"172.21.15.148:6812/2233349848","heartbeat_front_addr":"172.21.15.148:6811/2233349848","state":["exists","up"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2836839739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":2836839739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":2836839739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2836839739}]},"public_addr":"172.21.15.150:6808/2836839739","cluster_addr":"172.21.15.150:6809/2836839739","heartbeat_back_addr":"172.21.15.150:6811/2836839739","heartbeat_front_addr":"172.21.15.150:6810/2836839739","state":["exists","up"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4190376848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":4190376848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6807","nonce":4190376848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4190376848}]},"public_addr":"172.21.15.150:6804/4190376848","cluster_addr":"172.21.15.150:6805/4190376848","heartbeat_back_addr":"172.21.15.150:6807/4190376848","heartbeat_front_addr":"172.21.15.150:6806/4190376848","state":["exists","up"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2604225934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6801","nonce":2604225934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":2604225934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2604225934}]},"public_addr":"172.21.15.150:6800/2604225934","cluster_addr":"172.21.15.150:6801/2604225934","heartbeat_back_addr":"172.21.15.150:6803/2604225934","heartbeat_front_addr":"172.21.15.150:6802/2604225934","state":["exists","up"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3463178370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":3463178370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":3463178370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3463178370}]},"public_addr":"172.21.15.150:6812/3463178370","cluster_addr":"172.21.15.150:6813/3463178370","heartbeat_back_addr":"172.21.15.150:6815/3463178370","heartbeat_front_addr":"172.21.15.150:6814/3463178370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.920964+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.821114+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.365893+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:49.037497+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.184019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.644370+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:47.688638+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.429422+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:18:03.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 10 librados: watch_flush enter 2023-10-30T12:18:03.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 10 librados: watch_flush exit 2023-10-30T12:18:03.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 20 client.4278.objecter shutdown clearing up homeless session... 2023-10-30T12:18:03.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 10 client.4278.objecter successfully canceled tick 2023-10-30T12:18:03.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 1 -- 172.21.15.148:0/1164135969 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6c06c810 msgr2=0x7fab6c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 1 --2- 172.21.15.148:0/1164135969 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6c06c810 0x7fab6c06ecd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fab84000c00 tx=0x7fab8403e040 comp rx=0 tx=0).stop 2023-10-30T12:18:03.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 10 monclient: shutdown 2023-10-30T12:18:03.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 1 -- 172.21.15.148:0/1164135969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab8813e730 msgr2=0x7fab881c7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 1 --2- 172.21.15.148:0/1164135969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab8813e730 0x7fab881c7ce0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fab7800c3c0 tx=0x7fab7800c880 comp rx=0 tx=0).stop 2023-10-30T12:18:03.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 1 -- 172.21.15.148:0/1164135969 shutdown_connections 2023-10-30T12:18:03.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 1 --2- 172.21.15.148:0/1164135969 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6c06c810 0x7fab6c06ecd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 1 --2- 172.21.15.148:0/1164135969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab88141940 0x7fab881d6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 1 --2- 172.21.15.148:0/1164135969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab8813e730 0x7fab881c7ce0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 1 --2- 172.21.15.148:0/1164135969 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab880b7c10 0x7fab881c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.071+0000 7fab9047c640 1 -- 172.21.15.148:0/1164135969 >> 172.21.15.148:0/1164135969 conn(0x7fab8806c660 msgr2=0x7fab880b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.072+0000 7fab9047c640 1 -- 172.21.15.148:0/1164135969 shutdown_connections 2023-10-30T12:18:03.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.072+0000 7fab9047c640 1 -- 172.21.15.148:0/1164135969 wait complete. 2023-10-30T12:18:03.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.072+0000 7fab9047c640 1 librados: shutdown 2023-10-30T12:18:03.086 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T12:18:03.087 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T12:18:03.087 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T12:18:03.088 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T12:18:03.088 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T12:18:03.089 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T12:18:03.089 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-30T12:18:03.090 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-30T12:18:03.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.262+0000 7f331ed11640 10 monclient: get_monmap_and_config 2023-10-30T12:18:03.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.262+0000 7f331ed11640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.262+0000 7f331ed11640 10 monclient: monmap: 2023-10-30T12:18:03.265 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.265 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.266 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:03.263187+0000 2023-10-30T12:18:03.266 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:03.263187+0000 2023-10-30T12:18:03.266 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.266 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.267 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:03.267 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:03.267 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:03.267 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:03.267 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:03.267 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.263+0000 7f7ebdd47640 10 monclient: get_monmap_and_config 2023-10-30T12:18:03.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.263+0000 7f7ebdd47640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.263+0000 7f7ebdd47640 10 monclient: monmap: 2023-10-30T12:18:03.269 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.269 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.269 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:03.264426+0000 2023-10-30T12:18:03.270 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:03.264426+0000 2023-10-30T12:18:03.270 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.270 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.270 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:03.271 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:03.271 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:03.271 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:03.271 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:03.271 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.263+0000 7f331ed11640 1 Processor -- start 2023-10-30T12:18:03.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f7ebdd47640 1 Processor -- start 2023-10-30T12:18:03.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f7ebdd47640 1 -- start start 2023-10-30T12:18:03.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f7ebdd47640 10 monclient: init 2023-10-30T12:18:03.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.263+0000 7f331ed11640 1 -- start start 2023-10-30T12:18:03.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.263+0000 7f331ed11640 10 monclient: init 2023-10-30T12:18:03.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-10-30T12:18:03.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33180ae890 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:03.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3318149e10 0x7f331806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 10 monclient(hunting): picked mon.noname-c con 0x7f3318149e10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:03.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f331806d610 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:03.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3318071ad0 con 0x7f33180ae890 2023-10-30T12:18:03.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331806f7f0 con 0x7f331806d610 2023-10-30T12:18:03.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33180670d0 con 0x7f3318149e10 2023-10-30T12:18:03.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.264+0000 7f331ed11640 10 monclient(hunting): authenticate will time out at 1575.469521s 2023-10-30T12:18:03.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.265+0000 7f7ebdd47640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.265+0000 7f7ebdd47640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-10-30T12:18:03.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.265+0000 7f7ebdd47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7eb80acb50 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:03.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.265+0000 7f7ebdd47640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7eb806dd80 0x7f7eb8141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.265+0000 7f7ebdd47640 10 monclient(hunting): picked mon.noname-b con 0x7f7eb806dd80 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.265+0000 7f7ebdd47640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7eb806d6c0 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:03.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.265+0000 7f7ebdd47640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb8144be0 con 0x7f7eb80acb50 2023-10-30T12:18:03.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.266+0000 7f7ebdd47640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb8145b40 con 0x7f7eb806d6c0 2023-10-30T12:18:03.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.266+0000 7f7ebdd47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.266+0000 7f7ebdd47640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80670d0 con 0x7f7eb806dd80 2023-10-30T12:18:03.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.266+0000 7f7ebdd47640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.266+0000 7f7ebdd47640 10 monclient(hunting): authenticate will time out at 1575.471173s 2023-10-30T12:18:03.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.266+0000 7f7ebcd45640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7eb80acb50 0x7f7eb813eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52956/0 (socket says 172.21.15.148:52956) 2023-10-30T12:18:03.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.266+0000 7f7ebcd45640 1 -- 172.21.15.148:0/625532677 learned_addr learned my addr 172.21.15.148:0/625532677 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:18:03.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.266+0000 7f7eb7fff640 1 --2- 172.21.15.148:0/625532677 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7eb806dd80 0x7f7eb8141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f331d50e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3318149e10 0x7f331806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f331d50e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3318149e10 0x7f331806d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48626/0 (socket says 172.21.15.148:48626) 2023-10-30T12:18:03.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f331d50e640 1 -- 172.21.15.148:0/2854683924 learned_addr learned my addr 172.21.15.148:0/2854683924 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f331d50e640 10 monclient(hunting): get_auth_request con 0x7f3318149e10 auth_method 0 2023-10-30T12:18:03.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f331d50e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f331d50e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f331d50e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f7eb7fff640 10 monclient(hunting): get_auth_request con 0x7f7eb806dd80 auth_method 0 2023-10-30T12:18:03.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f7eb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f7eb7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f7eb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f331d50e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f331d50e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f331d50e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f9b9a973640 10 monclient: get_monmap_and_config 2023-10-30T12:18:03.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f9b9a973640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7f9b9a973640 10 monclient: monmap: 2023-10-30T12:18:03.287 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.287 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.287 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:03.268809+0000 2023-10-30T12:18:03.287 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:03.268809+0000 2023-10-30T12:18:03.287 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.288 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.288 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:03.288 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:03.288 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:03.288 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:03.289 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:03.289 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7fab84461640 10 monclient: get_monmap_and_config 2023-10-30T12:18:03.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7fab84461640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.267+0000 7fab84461640 10 monclient: monmap: 2023-10-30T12:18:03.290 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.290 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.290 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:03.268765+0000 2023-10-30T12:18:03.291 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:03.268765+0000 2023-10-30T12:18:03.291 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.291 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.291 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:03.291 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:03.291 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:03.292 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:03.292 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:03.292 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.268+0000 7f7eb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.268+0000 7f88f359e640 10 monclient: get_monmap_and_config 2023-10-30T12:18:03.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.268+0000 7f88f359e640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.268+0000 7f88f359e640 10 monclient: monmap: 2023-10-30T12:18:03.293 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.294 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.294 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:03.269529+0000 2023-10-30T12:18:03.294 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:03.269529+0000 2023-10-30T12:18:03.294 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.294 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.295 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:03.295 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:03.295 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:03.295 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:03.295 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:03.295 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.268+0000 7f7eb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.269+0000 7f7eb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.269+0000 7f9b9a973640 1 Processor -- start 2023-10-30T12:18:03.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.269+0000 7f9b9a973640 1 -- start start 2023-10-30T12:18:03.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.269+0000 7f9b9a973640 10 monclient: init 2023-10-30T12:18:03.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.269+0000 7f88f359e640 1 Processor -- start 2023-10-30T12:18:03.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.269+0000 7f88f359e640 1 -- start start 2023-10-30T12:18:03.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.269+0000 7f88f359e640 10 monclient: init 2023-10-30T12:18:03.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-10-30T12:18:03.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f88f406bc50 0x7f88f406af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 10 monclient(hunting): picked mon.noname-c con 0x7f88f406bc50 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:03.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88f4070550 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:03.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f88f4140d10 0x7f88f4141160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 10 monclient(hunting): picked mon.noname-b con 0x7f88f4140d10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f40709c0 con 0x7f88f4070550 2023-10-30T12:18:03.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f40670d0 con 0x7f88f406bc50 2023-10-30T12:18:03.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f41416a0 con 0x7f88f4140d10 2023-10-30T12:18:03.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f88f359e640 10 monclient(hunting): authenticate will time out at 1575.475430s 2023-10-30T12:18:03.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.269+0000 7fab84461640 1 Processor -- start 2023-10-30T12:18:03.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.269+0000 7fab84461640 1 -- start start 2023-10-30T12:18:03.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.269+0000 7fab84461640 10 monclient: init 2023-10-30T12:18:03.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-10-30T12:18:03.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab74034da0 0x7fab740c1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 10 monclient(hunting): picked mon.noname-c con 0x7fab74034da0 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:03.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab740c1750 0x7fab740c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 10 monclient(hunting): picked mon.noname-a con 0x7fab740c1750 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:03.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab7402a660 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:03.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7402cb00 con 0x7fab7402a660 2023-10-30T12:18:03.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab740087b0 con 0x7fab740c1750 2023-10-30T12:18:03.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab74033030 con 0x7fab74034da0 2023-10-30T12:18:03.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7fab84461640 10 monclient(hunting): authenticate will time out at 1575.475464s 2023-10-30T12:18:03.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-10-30T12:18:03.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9b940a9750 0x7f9b9413eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 10 monclient(hunting): picked mon.noname-c con 0x7f9b940a9750 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:03.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b9413f060 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:03.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f749c8e5640 10 monclient: get_monmap_and_config 2023-10-30T12:18:03.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f749c8e5640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b94142370 0x7f9b9414c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 10 monclient(hunting): picked mon.noname-a con 0x7f9b94142370 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:03.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b941417d0 con 0x7f9b9413f060 2023-10-30T12:18:03.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b940670d0 con 0x7f9b94142370 2023-10-30T12:18:03.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b940a9b30 con 0x7f9b940a9750 2023-10-30T12:18:03.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f9b9a973640 10 monclient(hunting): authenticate will time out at 1575.475045s 2023-10-30T12:18:03.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.270+0000 7f749c8e5640 10 monclient: monmap: 2023-10-30T12:18:03.312 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.312 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.312 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:03.271979+0000 2023-10-30T12:18:03.313 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:03.271979+0000 2023-10-30T12:18:03.313 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.313 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.313 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:03.313 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:03.313 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:03.314 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:03.314 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:03.314 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f331d50e640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2023-10-30T12:18:03.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f331d50e640 1 -- 172.21.15.148:0/2854683924 >> v1:172.21.15.150:6789/0 conn(0x7f331806d610 legacy=0x7f3318150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f331d50e640 1 -- 172.21.15.148:0/2854683924 >> v1:172.21.15.148:6789/0 conn(0x7f33180ae890 legacy=0x7f3318149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f331d50e640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f331d50e640 1 monclient: found mon.noname-c 2023-10-30T12:18:03.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f331d50e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f331d50e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f331d50e640 1 -- 172.21.15.148:0/2854683924 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331806fa60 con 0x7f3318149e10 2023-10-30T12:18:03.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f7eb7fff640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2023-10-30T12:18:03.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f7eb7fff640 1 -- 172.21.15.148:0/625532677 >> v1:172.21.15.150:6789/0 conn(0x7f7eb806d6c0 legacy=0x7f7eb8141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f7eb7fff640 1 -- 172.21.15.148:0/625532677 >> v1:172.21.15.148:6789/0 conn(0x7f7eb80acb50 legacy=0x7f7eb813eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f7eb7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f7eb7fff640 1 monclient: found mon.noname-b 2023-10-30T12:18:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f7eb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f7eb7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.271+0000 7f7eb7fff640 1 -- 172.21.15.148:0/625532677 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb806f070 con 0x7f7eb806dd80 2023-10-30T12:18:03.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.272+0000 7f749c8e5640 1 Processor -- start 2023-10-30T12:18:03.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.272+0000 7f749c8e5640 1 -- start start 2023-10-30T12:18:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.272+0000 7f749c8e5640 10 monclient: init 2023-10-30T12:18:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.272+0000 7f749c8e5640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.272+0000 7f749c8e5640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-10-30T12:18:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.272+0000 7f749c8e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f749806a030 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.272+0000 7f749c8e5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f74980ae9b0 0x7f74981419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.272+0000 7f749c8e5640 10 monclient(hunting): picked mon.noname-b con 0x7f74980ae9b0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.272+0000 7f749c8e5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7498141ee0 0x7f749814c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f749c8e5640 10 monclient(hunting): picked mon.noname-c con 0x7f7498141ee0 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f749c8e5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749806e3f0 con 0x7f749806a030 2023-10-30T12:18:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f749c8e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f749c8e5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7498067110 con 0x7f7498141ee0 2023-10-30T12:18:03.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f749c8e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f749c8e5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749806a410 con 0x7f74980ae9b0 2023-10-30T12:18:03.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f749c8e5640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f749c8e5640 10 monclient(hunting): authenticate will time out at 1575.477918s 2023-10-30T12:18:03.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f7eb7fff640 1 --2- 172.21.15.148:0/625532677 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7eb806dd80 0x7f7eb8141aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0097d0 tx=0x7f7eac02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af599eab0c1c1f7c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f88f1d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f88f4070550 0x7f88f413e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52964/0 (socket says 172.21.15.148:52964) 2023-10-30T12:18:03.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.273+0000 7f88f1d9b640 1 -- 172.21.15.148:0/3698726421 learned_addr learned my addr 172.21.15.148:0/3698726421 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:18:03.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.274+0000 7f9b98f10640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b94142370 0x7f9b9414c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.274+0000 7f59e44a2640 10 monclient: get_monmap_and_config 2023-10-30T12:18:03.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.274+0000 7f59e44a2640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.274+0000 7f59e44a2640 10 monclient: monmap: 2023-10-30T12:18:03.326 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.326 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.327 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:03.275631+0000 2023-10-30T12:18:03.327 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:03.275631+0000 2023-10-30T12:18:03.327 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.327 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.327 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:03.328 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:03.328 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:03.328 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:03.328 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:03.328 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.275+0000 7f59e44a2640 1 Processor -- start 2023-10-30T12:18:03.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.275+0000 7f59e44a2640 1 -- start start 2023-10-30T12:18:03.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.275+0000 7f59e44a2640 10 monclient: init 2023-10-30T12:18:03.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.275+0000 7fab819fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab740c1750 0x7fab740c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.275+0000 7fab819fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab740c1750 0x7fab740c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59804/0 (socket says 172.21.15.148:59804) 2023-10-30T12:18:03.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.275+0000 7fab819fc640 1 -- 172.21.15.148:0/2026037036 learned_addr learned my addr 172.21.15.148:0/2026037036 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.275+0000 7fab819fc640 10 monclient(hunting): get_auth_request con 0x7fab740c1750 auth_method 0 2023-10-30T12:18:03.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.275+0000 7fab819fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.275+0000 7fab819fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.275+0000 7fab819fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f88f2d9d640 1 --2- 172.21.15.148:0/3698726421 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f88f4140d10 0x7f88f4141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-10-30T12:18:03.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59dc0b7c10 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:03.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59dc141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:03.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59dc14a460 0x7f59dc14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 10 monclient(hunting): picked mon.noname-c con 0x7f59dc14a460 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:03.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59dc0a9350 con 0x7f59dc0b7c10 2023-10-30T12:18:03.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59dc0aa6a0 con 0x7f59dc141940 2023-10-30T12:18:03.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59dc0670a0 con 0x7f59dc14a460 2023-10-30T12:18:03.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f59e44a2640 10 monclient(hunting): authenticate will time out at 1575.481319s 2023-10-30T12:18:03.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.277+0000 7f331d50e640 1 --2- 172.21.15.148:0/2854683924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3318149e10 0x7f331806d0d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3314009870 tx=0x7f331402b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6b251c5ef67b402 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f9b98f10640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b94142370 0x7f9b9414c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59792/0 (socket says 172.21.15.148:59792) 2023-10-30T12:18:03.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f9b98f10640 1 -- 172.21.15.148:0/2325458155 learned_addr learned my addr 172.21.15.148:0/2325458155 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f9b98f10640 10 monclient(hunting): get_auth_request con 0x7f9b94142370 auth_method 0 2023-10-30T12:18:03.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f9b98f10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f9b98f10640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.276+0000 7f9b98f10640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.277+0000 7f9b93fff640 1 --2- 172.21.15.148:0/2325458155 >> v2:172.21.15.150:3300/0 conn(0x7f9b940a9750 0x7f9b9413eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.277+0000 7fab819fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.277+0000 7fab819fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.277+0000 7fab819fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.277+0000 7f9b98f10640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.277+0000 7f9b98f10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.277+0000 7f9b98f10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.278+0000 7f88f2d9d640 10 monclient(hunting): get_auth_request con 0x7f88f4140d10 auth_method 0 2023-10-30T12:18:03.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.278+0000 7f88f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.278+0000 7f88f2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.278+0000 7f88f2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.278+0000 7f9b93fff640 10 monclient(hunting): get_auth_request con 0x7f9b940a9750 auth_method 0 2023-10-30T12:18:03.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.278+0000 7f9b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.278+0000 7f9b93fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.278+0000 7f9b93fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7496ffd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f74980ae9b0 0x7f74981419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f88f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f88f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f88f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b98f10640 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2023-10-30T12:18:03.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b98f10640 1 -- 172.21.15.148:0/2325458155 >> v2:172.21.15.150:3300/0 conn(0x7f9b940a9750 msgr2=0x7f9b9413eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:03.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b98f10640 1 --2- 172.21.15.148:0/2325458155 >> v2:172.21.15.150:3300/0 conn(0x7f9b940a9750 0x7f9b9413eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b98f10640 1 -- 172.21.15.148:0/2325458155 >> v1:172.21.15.150:6789/0 conn(0x7f9b9413f060 legacy=0x7f9b94141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:03.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b98f10640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b98f10640 1 monclient: found mon.noname-a 2023-10-30T12:18:03.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b98f10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b98f10640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:03.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f9b98f10640 1 -- 172.21.15.148:0/2325458155 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b940aa5f0 con 0x7f9b94142370 2023-10-30T12:18:03.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 1 -- 172.21.15.148:0/625532677 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac031770 con 0x7f7eb806dd80 2023-10-30T12:18:03.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:03.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 10 monclient: dump: 2023-10-30T12:18:03.349 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.349 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.349 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.350 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.350 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.350 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.350 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.350 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.351 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.351 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 1 -- 172.21.15.148:0/625532677 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac0318e0 con 0x7f7eb806dd80 2023-10-30T12:18:03.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 1 -- 172.21.15.148:0/625532677 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac031c80 con 0x7f7eb806dd80 2023-10-30T12:18:03.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:03.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7eb77fe640 10 monclient: dump: 2023-10-30T12:18:03.353 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.353 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.353 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.353 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.353 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.354 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.354 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.354 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.354 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.354 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7ebdd47640 5 monclient: authenticate success, global_id 4241 2023-10-30T12:18:03.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7ebdd47640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:03.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7f7ebdd47640 4 set_mon_vals no callback set 2023-10-30T12:18:03.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7fccbed67640 10 monclient: get_monmap_and_config 2023-10-30T12:18:03.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7fccbed67640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.279+0000 7fccbed67640 10 monclient: monmap: 2023-10-30T12:18:03.356 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.356 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.357 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:03.280983+0000 2023-10-30T12:18:03.357 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:03.280983+0000 2023-10-30T12:18:03.357 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.357 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.357 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:03.358 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:03.358 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:03.358 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:03.358 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:03.358 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.280+0000 7fab821fd640 1 --2- 172.21.15.148:0/2026037036 >> v2:172.21.15.150:3300/0 conn(0x7fab74034da0 0x7fab740c1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.280+0000 7f7ebdd47640 10 monclient: shutdown 2023-10-30T12:18:03.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.280+0000 7f7ebdd47640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.280+0000 7f7ebdd47640 1 -- 172.21.15.148:0/625532677 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7eb806dd80 msgr2=0x7f7eb8141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.280+0000 7f7ebdd47640 1 --2- 172.21.15.148:0/625532677 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7eb806dd80 0x7f7eb8141aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0097d0 tx=0x7f7eac02b810 comp rx=0 tx=0).stop 2023-10-30T12:18:03.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.280+0000 7fccbed67640 1 Processor -- start 2023-10-30T12:18:03.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.280+0000 7f7ebdd47640 1 -- 172.21.15.148:0/625532677 shutdown_connections 2023-10-30T12:18:03.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.280+0000 7f7ebdd47640 1 --2- 172.21.15.148:0/625532677 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7eb806dd80 0x7f7eb8141aa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.280+0000 7f7ebdd47640 1 -- 172.21.15.148:0/625532677 >> 172.21.15.148:0/625532677 conn(0x7f7eb8069dc0 msgr2=0x7f7eb806a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7f7ebdd47640 1 -- 172.21.15.148:0/625532677 shutdown_connections 2023-10-30T12:18:03.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 1 -- start start 2023-10-30T12:18:03.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient: init 2023-10-30T12:18:03.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7f7ebdd47640 1 -- 172.21.15.148:0/625532677 wait complete. 2023-10-30T12:18:03.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-10-30T12:18:03.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb0039820 0x7fccb0033890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient(hunting): picked mon.noname-b con 0x7fccb0039820 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fccb00364b0 0x7fccb0033dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient(hunting): picked mon.noname-a con 0x7fccb00364b0 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:03.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fccb00c7db0 0x7fccb00c8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient(hunting): picked mon.noname-c con 0x7fccb00c7db0 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:03.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb0008710 con 0x7fccb00364b0 2023-10-30T12:18:03.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb0036060 con 0x7fccb00c7db0 2023-10-30T12:18:03.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0039c00 con 0x7fccb0039820 2023-10-30T12:18:03.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7fccbed67640 10 monclient(hunting): authenticate will time out at 1575.486671s 2023-10-30T12:18:03.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7f7ebdd47640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.281+0000 7f7ebdd47640 10 monclient: monmap: 2023-10-30T12:18:03.367 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.367 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:03.367 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:03.368 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:03.368 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.368 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:03.368 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:03.368 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:03.369 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:03.369 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f7ebdd47640 1 librados: starting msgr at 2023-10-30T12:18:03.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f88f2d9d640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2023-10-30T12:18:03.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f88f2d9d640 1 -- 172.21.15.148:0/3698726421 >> v2:172.21.15.150:3300/0 conn(0x7f88f406bc50 msgr2=0x7f88f406af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:03.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f88f2d9d640 1 --2- 172.21.15.148:0/3698726421 >> v2:172.21.15.150:3300/0 conn(0x7f88f406bc50 0x7f88f406af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f88f2d9d640 1 -- 172.21.15.148:0/3698726421 >> v1:172.21.15.148:6789/0 conn(0x7f88f4070550 legacy=0x7f88f413e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f88f2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f88f2d9d640 1 monclient: found mon.noname-b 2023-10-30T12:18:03.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f88f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f88f2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f88f2d9d640 1 -- 172.21.15.148:0/3698726421 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f4071940 con 0x7f88f4140d10 2023-10-30T12:18:03.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f7ebdd47640 1 librados: starting objecter 2023-10-30T12:18:03.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f7ebdd47640 1 Processor -- start 2023-10-30T12:18:03.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f7ebdd47640 1 -- start start 2023-10-30T12:18:03.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f7ebdd47640 1 librados: setting wanted keys 2023-10-30T12:18:03.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f7ebdd47640 1 librados: calling monclient init 2023-10-30T12:18:03.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.282+0000 7f7ebdd47640 10 monclient: init 2023-10-30T12:18:03.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7496ffd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f74980ae9b0 0x7f74981419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54406/0 (socket says 172.21.15.148:54406) 2023-10-30T12:18:03.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7496ffd640 1 -- 172.21.15.148:0/3891264078 learned_addr learned my addr 172.21.15.148:0/3891264078 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7496ffd640 10 monclient(hunting): get_auth_request con 0x7f74980ae9b0 auth_method 0 2023-10-30T12:18:03.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7496ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7496ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7496ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:18:03.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb806d6c0 0x7f7eb813a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 10 monclient(hunting): picked mon.noname-a con 0x7f7eb806d6c0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:03.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806dd80 0x7f7eb8135f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 10 monclient(hunting): picked mon.noname-b con 0x7f7eb806dd80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:03.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7eb80acb50 0x7f7eb81364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 10 monclient(hunting): picked mon.noname-c con 0x7f7eb80acb50 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb806b2b0 con 0x7f7eb806d6c0 2023-10-30T12:18:03.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb8070b10 con 0x7f7eb806dd80 2023-10-30T12:18:03.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb8144530 con 0x7f7eb80acb50 2023-10-30T12:18:03.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.283+0000 7f7ebdd47640 10 monclient(hunting): authenticate will time out at 1575.488479s 2023-10-30T12:18:03.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.284+0000 7f7496ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.284+0000 7f7496ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.284+0000 7f7496ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7fab819fc640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2023-10-30T12:18:03.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7fab819fc640 1 -- 172.21.15.148:0/2026037036 >> v2:172.21.15.150:3300/0 conn(0x7fab74034da0 msgr2=0x7fab740c1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:03.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7fab819fc640 1 --2- 172.21.15.148:0/2026037036 >> v2:172.21.15.150:3300/0 conn(0x7fab74034da0 0x7fab740c1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7fab819fc640 1 -- 172.21.15.148:0/2026037036 >> v1:172.21.15.148:6789/0 conn(0x7fab7402a660 legacy=0x7fab740292d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7fab819fc640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7fab819fc640 1 monclient: found mon.noname-a 2023-10-30T12:18:03.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7fab819fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7fab819fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:03.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7fab819fc640 1 -- 172.21.15.148:0/2026037036 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7402b5c0 con 0x7fab740c1750 2023-10-30T12:18:03.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7f59e2a3f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59dc14a460 0x7f59dc14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.285+0000 7f88f2d9d640 1 --2- 172.21.15.148:0/3698726421 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f88f4140d10 0x7f88f4141160 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009d90 tx=0x7f88ec02bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92a65f7c86f6b175 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f59e2a3f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59dc14a460 0x7f59dc14c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48658/0 (socket says 172.21.15.148:48658) 2023-10-30T12:18:03.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f59e2a3f640 1 -- 172.21.15.148:0/1457405758 learned_addr learned my addr 172.21.15.148:0/1457405758 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f59e2a3f640 10 monclient(hunting): get_auth_request con 0x7f59dc14a460 auth_method 0 2023-10-30T12:18:03.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f59e2a3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f59e2a3f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f59e2a3f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f7496ffd640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2023-10-30T12:18:03.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f7496ffd640 1 -- 172.21.15.148:0/3891264078 >> v2:172.21.15.150:3300/0 conn(0x7f7498141ee0 msgr2=0x7f749814c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:03.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f7496ffd640 1 --2- 172.21.15.148:0/3891264078 >> v2:172.21.15.150:3300/0 conn(0x7f7498141ee0 0x7f749814c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f7496ffd640 1 -- 172.21.15.148:0/3891264078 >> v1:172.21.15.148:6789/0 conn(0x7f749806a030 legacy=0x7f749813ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f7496ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f7496ffd640 1 monclient: found mon.noname-b 2023-10-30T12:18:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f7496ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f7496ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f7496ffd640 1 -- 172.21.15.148:0/3891264078 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74980af110 con 0x7f74980ae9b0 2023-10-30T12:18:03.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f59e2a3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f59e2a3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.287+0000 7f59e2a3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.288+0000 7f9b98f10640 1 --2- 172.21.15.148:0/2325458155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b94142370 0x7f9b9414c0e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9b9406ba50 tx=0x7f9b8802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ca5db40e73e4bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.290+0000 7f59e2a3f640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2023-10-30T12:18:03.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.290+0000 7f59e2a3f640 1 -- 172.21.15.148:0/1457405758 >> v1:172.21.15.150:6789/0 conn(0x7f59dc141940 legacy=0x7f59dc141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:03.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.290+0000 7f59e2a3f640 1 -- 172.21.15.148:0/1457405758 >> v1:172.21.15.148:6789/0 conn(0x7f59dc0b7c10 legacy=0x7f59dc0a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:03.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.290+0000 7f59e2a3f640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.290+0000 7f59e2a3f640 1 monclient: found mon.noname-c 2023-10-30T12:18:03.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.290+0000 7f59e2a3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.290+0000 7f59e2a3f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.290+0000 7f59e2a3f640 1 -- 172.21.15.148:0/1457405758 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59dc0aa910 con 0x7f59dc14a460 2023-10-30T12:18:03.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 1 -- 172.21.15.148:0/2854683924 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314030850 con 0x7f3318149e10 2023-10-30T12:18:03.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:03.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 10 monclient: dump: 2023-10-30T12:18:03.394 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.394 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.394 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.395 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.395 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.395 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.395 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.395 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.396 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.396 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 1 -- 172.21.15.148:0/2854683924 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33140309c0 con 0x7f3318149e10 2023-10-30T12:18:03.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331ed11640 5 monclient: authenticate success, global_id 4303 2023-10-30T12:18:03.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 1 -- 172.21.15.148:0/2854683924 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314030d60 con 0x7f3318149e10 2023-10-30T12:18:03.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331ed11640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:03.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331ed11640 4 set_mon_vals no callback set 2023-10-30T12:18:03.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:03.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.292+0000 7f331cd0d640 10 monclient: dump: 2023-10-30T12:18:03.398 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.399 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.399 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.399 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.399 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.399 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.400 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.400 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.400 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.400 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.293+0000 7f59e2a3f640 1 --2- 172.21.15.148:0/1457405758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc14a460 0x7f59dc14c850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009e50 tx=0x7f59d402be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f23dc56221dffef1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.293+0000 7f331ed11640 10 monclient: shutdown 2023-10-30T12:18:03.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.293+0000 7f331ed11640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.293+0000 7f331ed11640 1 -- 172.21.15.148:0/2854683924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3318149e10 msgr2=0x7f331806d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.293+0000 7f331ed11640 1 --2- 172.21.15.148:0/2854683924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3318149e10 0x7f331806d0d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3314009870 tx=0x7f331402b990 comp rx=0 tx=0).stop 2023-10-30T12:18:03.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.293+0000 7f331ed11640 1 -- 172.21.15.148:0/2854683924 shutdown_connections 2023-10-30T12:18:03.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.293+0000 7f331ed11640 1 --2- 172.21.15.148:0/2854683924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3318149e10 0x7f331806d0d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.293+0000 7f331ed11640 1 -- 172.21.15.148:0/2854683924 >> 172.21.15.148:0/2854683924 conn(0x7f3318069dc0 msgr2=0x7f331806a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.293+0000 7f331ed11640 1 -- 172.21.15.148:0/2854683924 shutdown_connections 2023-10-30T12:18:03.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.293+0000 7f331ed11640 1 -- 172.21.15.148:0/2854683924 wait complete. 2023-10-30T12:18:03.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7fccbdd65640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb0039820 0x7fccb0033890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f331ed11640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7fab819fc640 1 --2- 172.21.15.148:0/2026037036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab740c1750 0x7fab740c7370 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fab74007390 tx=0x7fab6c02fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26498fbe35d2717e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f331ed11640 10 monclient: monmap: 2023-10-30T12:18:03.405 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.406 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:03.406 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:03.406 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:03.406 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.407 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:03.407 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:03.407 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:03.407 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:03.408 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 1 -- 172.21.15.148:0/3698726421 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec02cce0 con 0x7f88f4140d10 2023-10-30T12:18:03.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:03.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 10 monclient: dump: 2023-10-30T12:18:03.409 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.410 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.410 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.410 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.411 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.411 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.411 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.411 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.411 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.412 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 1 -- 172.21.15.148:0/3698726421 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec02ce50 con 0x7f88f4140d10 2023-10-30T12:18:03.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f359e640 5 monclient: authenticate success, global_id 4244 2023-10-30T12:18:03.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:03.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f359e640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:03.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f359e640 4 set_mon_vals no callback set 2023-10-30T12:18:03.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 1 -- 172.21.15.148:0/3698726421 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec034660 con 0x7f88f4140d10 2023-10-30T12:18:03.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:03.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f88f159a640 10 monclient: dump: 2023-10-30T12:18:03.415 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.415 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.415 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.415 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.415 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.416 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.416 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.416 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.417 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.417 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f331ed11640 1 librados: starting msgr at 2023-10-30T12:18:03.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f331ed11640 1 librados: starting objecter 2023-10-30T12:18:03.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7fccbdd65640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb0039820 0x7fccb0033890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54412/0 (socket says 172.21.15.148:54412) 2023-10-30T12:18:03.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7fccbdd65640 1 -- 172.21.15.148:0/1127019227 learned_addr learned my addr 172.21.15.148:0/1127019227 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7fccbdd65640 10 monclient(hunting): get_auth_request con 0x7fccb0039820 auth_method 0 2023-10-30T12:18:03.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7fccbdd65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7fccbdd65640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7fccbdd65640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f331ed11640 1 Processor -- start 2023-10-30T12:18:03.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f331ed11640 1 -- start start 2023-10-30T12:18:03.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f331ed11640 1 librados: setting wanted keys 2023-10-30T12:18:03.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f331ed11640 1 librados: calling monclient init 2023-10-30T12:18:03.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.294+0000 7f331ed11640 10 monclient: init 2023-10-30T12:18:03.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f7496ffd640 1 --2- 172.21.15.148:0/3891264078 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f74980ae9b0 0x7f74981419a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f748c009870 tx=0x7f748c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa7d7180795a4cb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:18:03.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f88f359e640 10 monclient: shutdown 2023-10-30T12:18:03.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f88f359e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f88f359e640 1 -- 172.21.15.148:0/3698726421 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f88f4140d10 msgr2=0x7f88f4141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f88f359e640 1 --2- 172.21.15.148:0/3698726421 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f88f4140d10 0x7f88f4141160 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009d90 tx=0x7f88ec02bf30 comp rx=0 tx=0).stop 2023-10-30T12:18:03.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f331806d610 0x7f33180ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 10 monclient(hunting): picked mon.noname-a con 0x7f331806d610 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:03.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33180ae890 0x7f33181263b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 10 monclient(hunting): picked mon.noname-b con 0x7f33180ae890 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:03.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3318149e10 0x7f33181dab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 10 monclient(hunting): picked mon.noname-c con 0x7f3318149e10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331806d0d0 con 0x7f331806d610 2023-10-30T12:18:03.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3318125f20 con 0x7f33180ae890 2023-10-30T12:18:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3318124530 con 0x7f3318149e10 2023-10-30T12:18:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f331ed11640 10 monclient(hunting): authenticate will time out at 1575.500472s 2023-10-30T12:18:03.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f88f359e640 1 -- 172.21.15.148:0/3698726421 shutdown_connections 2023-10-30T12:18:03.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f88f359e640 1 --2- 172.21.15.148:0/3698726421 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f88f4140d10 0x7f88f4141160 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f88f359e640 1 --2- 172.21.15.148:0/3698726421 >> v2:172.21.15.150:3300/0 conn(0x7f88f406bc50 0x7f88f406af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f88f359e640 1 -- 172.21.15.148:0/3698726421 >> 172.21.15.148:0/3698726421 conn(0x7f88f4069dc0 msgr2=0x7f88f406a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7fccbdd65640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7fccbdd65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7fccbdd65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f88f359e640 1 -- 172.21.15.148:0/3698726421 shutdown_connections 2023-10-30T12:18:03.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.295+0000 7f88f359e640 1 -- 172.21.15.148:0/3698726421 wait complete. 2023-10-30T12:18:03.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.296+0000 7f88f359e640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.296+0000 7f88f359e640 10 monclient: monmap: 2023-10-30T12:18:03.432 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.432 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:03.432 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:03.433 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:03.433 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.433 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:03.433 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:03.433 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:03.434 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:03.434 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.296+0000 7f88f359e640 1 librados: starting msgr at 2023-10-30T12:18:03.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.296+0000 7f88f359e640 1 librados: starting objecter 2023-10-30T12:18:03.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 1 Processor -- start 2023-10-30T12:18:03.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 1 -- start start 2023-10-30T12:18:03.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 1 librados: setting wanted keys 2023-10-30T12:18:03.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 1 librados: calling monclient init 2023-10-30T12:18:03.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient: init 2023-10-30T12:18:03.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:03.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f406bc50 0x7f88f41360b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient(hunting): picked mon.noname-a con 0x7f88f406bc50 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:03.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f88f4070550 0x7f88f40ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient(hunting): picked mon.noname-c con 0x7f88f4070550 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88f4140d10 0x7f88f4137e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient(hunting): picked mon.noname-b con 0x7f88f4140d10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:03.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f40aca30 con 0x7f88f406bc50 2023-10-30T12:18:03.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f4137800 con 0x7f88f4140d10 2023-10-30T12:18:03.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f4136740 con 0x7f88f4070550 2023-10-30T12:18:03.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f88f359e640 10 monclient(hunting): authenticate will time out at 1575.502717s 2023-10-30T12:18:03.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.297+0000 7f7eb7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806dd80 0x7f7eb8135f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.298+0000 7fccbdd65640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2023-10-30T12:18:03.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.298+0000 7fccbdd65640 1 -- 172.21.15.148:0/1127019227 >> v2:172.21.15.150:3300/0 conn(0x7fccb00c7db0 msgr2=0x7fccb00c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:03.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.298+0000 7fccbdd65640 1 --2- 172.21.15.148:0/1127019227 >> v2:172.21.15.150:3300/0 conn(0x7fccb00c7db0 0x7fccb00c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.298+0000 7fccbdd65640 1 -- 172.21.15.148:0/1127019227 >> v2:172.21.15.148:3300/0 conn(0x7fccb00364b0 msgr2=0x7fccb0033dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:03.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.298+0000 7fccbdd65640 1 --2- 172.21.15.148:0/1127019227 >> v2:172.21.15.148:3300/0 conn(0x7fccb00364b0 0x7fccb0033dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.298+0000 7fccbdd65640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.298+0000 7fccbdd65640 1 monclient: found mon.noname-b 2023-10-30T12:18:03.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.298+0000 7fccbdd65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.298+0000 7fccbdd65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.298+0000 7fccbdd65640 1 -- 172.21.15.148:0/1127019227 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb00c5710 con 0x7fccb0039820 2023-10-30T12:18:03.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.301+0000 7fccbdd65640 1 --2- 172.21.15.148:0/1127019227 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb0039820 0x7fccb0033890 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009c50 tx=0x7fccb402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27713cf758eeb027 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f7eb7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806dd80 0x7f7eb8135f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48688/0 (socket says 172.21.15.148:48688) 2023-10-30T12:18:03.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f7eb7fff640 1 -- 172.21.15.148:0/3037561332 learned_addr learned my addr 172.21.15.148:0/3037561332 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f7eb7fff640 10 monclient(hunting): get_auth_request con 0x7f7eb806dd80 auth_method 0 2023-10-30T12:18:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f7eb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f7eb7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f7eb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 1 -- 172.21.15.148:0/2325458155 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b880317e0 con 0x7f9b94142370 2023-10-30T12:18:03.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:03.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 10 monclient: dump: 2023-10-30T12:18:03.446 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.447 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.447 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.447 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.448 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.448 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.448 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.448 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.448 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.449 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 1 -- 172.21.15.148:0/2325458155 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b88031950 con 0x7f9b94142370 2023-10-30T12:18:03.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 1 -- 172.21.15.148:0/2325458155 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88031cf0 con 0x7f9b94142370 2023-10-30T12:18:03.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:03.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b92ffd640 10 monclient: dump: 2023-10-30T12:18:03.452 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.452 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.453 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.453 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.453 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.453 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.454 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.454 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.454 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.454 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b9a973640 5 monclient: authenticate success, global_id 4287 2023-10-30T12:18:03.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b9a973640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:03.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.303+0000 7f9b9a973640 4 set_mon_vals no callback set 2023-10-30T12:18:03.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f9b9a973640 10 monclient: shutdown 2023-10-30T12:18:03.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f9b9a973640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f9b9a973640 1 -- 172.21.15.148:0/2325458155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b94142370 msgr2=0x7f9b9414c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f9b9a973640 1 --2- 172.21.15.148:0/2325458155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b94142370 0x7f9b9414c0e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9b9406ba50 tx=0x7f9b8802b880 comp rx=0 tx=0).stop 2023-10-30T12:18:03.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f9b9a973640 1 -- 172.21.15.148:0/2325458155 shutdown_connections 2023-10-30T12:18:03.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f9b9a973640 1 --2- 172.21.15.148:0/2325458155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b94142370 0x7f9b9414c0e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f9b9a973640 1 --2- 172.21.15.148:0/2325458155 >> v2:172.21.15.150:3300/0 conn(0x7f9b940a9750 0x7f9b9413eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f9b9a973640 1 -- 172.21.15.148:0/2325458155 >> 172.21.15.148:0/2325458155 conn(0x7f9b94069dc0 msgr2=0x7f9b9406a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f9b9a973640 1 -- 172.21.15.148:0/2325458155 shutdown_connections 2023-10-30T12:18:03.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f7ebd546640 1 --2- 172.21.15.148:0/3037561332 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7eb80acb50 0x7f7eb81364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.304+0000 7f9b9a973640 1 -- 172.21.15.148:0/2325458155 wait complete. 2023-10-30T12:18:03.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f9b9a973640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f9b9a973640 10 monclient: monmap: 2023-10-30T12:18:03.458 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.458 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:03.459 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:03.459 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:03.462 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.462 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:03.462 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:03.462 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:03.462 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:03.463 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f331dd0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f331806d610 0x7f33180ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f9b9a973640 1 librados: starting msgr at 2023-10-30T12:18:03.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f9b9a973640 1 librados: starting objecter 2023-10-30T12:18:03.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 1 -- 172.21.15.148:0/3891264078 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c0317c0 con 0x7f74980ae9b0 2023-10-30T12:18:03.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 10 monclient: dump: 2023-10-30T12:18:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.466 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.466 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.466 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.466 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.467 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.467 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f9b9a973640 1 Processor -- start 2023-10-30T12:18:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 1 -- 172.21.15.148:0/3891264078 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748c02e070 con 0x7f74980ae9b0 2023-10-30T12:18:03.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 1 -- 172.21.15.148:0/3891264078 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c031b50 con 0x7f74980ae9b0 2023-10-30T12:18:03.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:03.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f74967fc640 10 monclient: dump: 2023-10-30T12:18:03.470 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.470 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.470 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.470 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.470 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.471 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.471 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.471 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.471 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.471 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f749c8e5640 5 monclient: authenticate success, global_id 4247 2023-10-30T12:18:03.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f749c8e5640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:03.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f749c8e5640 4 set_mon_vals no callback set 2023-10-30T12:18:03.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 1 -- 172.21.15.148:0/1457405758 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d402c9f0 con 0x7f59dc14a460 2023-10-30T12:18:03.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:03.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 10 monclient: dump: 2023-10-30T12:18:03.474 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.474 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.474 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.474 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.474 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.475 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.475 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.475 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.475 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.475 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 1 -- 172.21.15.148:0/1457405758 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d402cb60 con 0x7f59dc14a460 2023-10-30T12:18:03.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 1 -- 172.21.15.148:0/1457405758 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d40343f0 con 0x7f59dc14a460 2023-10-30T12:18:03.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:03.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e123c640 10 monclient: dump: 2023-10-30T12:18:03.477 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.478 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.478 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.478 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.479 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.479 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.479 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.479 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.479 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.480 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f331dd0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f331806d610 0x7f33180ab7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59824/0 (socket says 172.21.15.148:59824) 2023-10-30T12:18:03.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f331dd0f640 1 -- 172.21.15.148:0/1915954374 learned_addr learned my addr 172.21.15.148:0/1915954374 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f331dd0f640 10 monclient(hunting): get_auth_request con 0x7f331806d610 auth_method 0 2023-10-30T12:18:03.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f331dd0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f331dd0f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f331dd0f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e44a2640 5 monclient: authenticate success, global_id 4312 2023-10-30T12:18:03.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e44a2640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:03.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f59e44a2640 4 set_mon_vals no callback set 2023-10-30T12:18:03.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f9b9a973640 1 -- start start 2023-10-30T12:18:03.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f9b9a973640 1 librados: setting wanted keys 2023-10-30T12:18:03.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f9b9a973640 1 librados: calling monclient init 2023-10-30T12:18:03.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.305+0000 7f9b9a973640 10 monclient: init 2023-10-30T12:18:03.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f7ebd546640 10 monclient(hunting): get_auth_request con 0x7f7eb80acb50 auth_method 0 2023-10-30T12:18:03.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f7ebd546640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f7ebd546640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f7ebd546640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f88f259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f406bc50 0x7f88f41360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:03.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b940a9750 0x7f9b941cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 10 monclient(hunting): picked mon.noname-b con 0x7f9b940a9750 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:03.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9b9413f060 0x7f9b941d6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f331dd0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f331dd0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f331dd0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f59e44a2640 10 monclient: shutdown 2023-10-30T12:18:03.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f59e44a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f59e44a2640 1 -- 172.21.15.148:0/1457405758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc14a460 msgr2=0x7f59dc14c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f59e44a2640 1 --2- 172.21.15.148:0/1457405758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc14a460 0x7f59dc14c850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009e50 tx=0x7f59d402be50 comp rx=0 tx=0).stop 2023-10-30T12:18:03.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f749c8e5640 10 monclient: shutdown 2023-10-30T12:18:03.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f749c8e5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f749c8e5640 1 -- 172.21.15.148:0/3891264078 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f74980ae9b0 msgr2=0x7f74981419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f749c8e5640 1 --2- 172.21.15.148:0/3891264078 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f74980ae9b0 0x7f74981419a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f748c009870 tx=0x7f748c02b860 comp rx=0 tx=0).stop 2023-10-30T12:18:03.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f749c8e5640 1 -- 172.21.15.148:0/3891264078 shutdown_connections 2023-10-30T12:18:03.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f749c8e5640 1 --2- 172.21.15.148:0/3891264078 >> v2:172.21.15.150:3300/0 conn(0x7f7498141ee0 0x7f749814c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f749c8e5640 1 --2- 172.21.15.148:0/3891264078 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f74980ae9b0 0x7f74981419a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 10 monclient(hunting): picked mon.noname-c con 0x7f9b9413f060 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b94142370 0x7f9b941d7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 10 monclient(hunting): picked mon.noname-a con 0x7f9b94142370 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:03.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940ad840 con 0x7f9b94142370 2023-10-30T12:18:03.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b941ce7d0 con 0x7f9b940a9750 2023-10-30T12:18:03.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b94145570 con 0x7f9b9413f060 2023-10-30T12:18:03.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f9b9a973640 10 monclient(hunting): authenticate will time out at 1575.511437s 2023-10-30T12:18:03.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f749c8e5640 1 -- 172.21.15.148:0/3891264078 >> 172.21.15.148:0/3891264078 conn(0x7f74980b41e0 msgr2=0x7f74980b45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f7eb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f7eb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f7eb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f749c8e5640 1 -- 172.21.15.148:0/3891264078 shutdown_connections 2023-10-30T12:18:03.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f749c8e5640 1 -- 172.21.15.148:0/3891264078 wait complete. 2023-10-30T12:18:03.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f59e44a2640 1 -- 172.21.15.148:0/1457405758 shutdown_connections 2023-10-30T12:18:03.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f59e44a2640 1 --2- 172.21.15.148:0/1457405758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc14a460 0x7f59dc14c850 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f59e44a2640 1 -- 172.21.15.148:0/1457405758 >> 172.21.15.148:0/1457405758 conn(0x7f59dc06c660 msgr2=0x7f59dc0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f59e44a2640 1 -- 172.21.15.148:0/1457405758 shutdown_connections 2023-10-30T12:18:03.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.306+0000 7f59e44a2640 1 -- 172.21.15.148:0/1457405758 wait complete. 2023-10-30T12:18:03.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.307+0000 7f59e44a2640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.307+0000 7f59e44a2640 10 monclient: monmap: 2023-10-30T12:18:03.499 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.499 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:03.499 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:03.499 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:03.500 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.500 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:03.500 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:03.500 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:03.500 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:03.501 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.307+0000 7f749c8e5640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.307+0000 7f749c8e5640 10 monclient: monmap: 2023-10-30T12:18:03.501 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.501 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:03.502 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:03.502 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:03.502 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.502 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:03.503 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:03.503 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:03.503 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:03.503 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.307+0000 7f59e44a2640 1 librados: starting msgr at 2023-10-30T12:18:03.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.307+0000 7f59e44a2640 1 librados: starting objecter 2023-10-30T12:18:03.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.307+0000 7f749c8e5640 1 librados: starting msgr at 2023-10-30T12:18:03.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.307+0000 7f749c8e5640 1 librados: starting objecter 2023-10-30T12:18:03.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.307+0000 7f59e44a2640 1 Processor -- start 2023-10-30T12:18:03.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.307+0000 7f749c8e5640 1 Processor -- start 2023-10-30T12:18:03.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 1 -- start start 2023-10-30T12:18:03.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 1 librados: setting wanted keys 2023-10-30T12:18:03.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 1 librados: calling monclient init 2023-10-30T12:18:03.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient: init 2023-10-30T12:18:03.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 1 -- start start 2023-10-30T12:18:03.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 1 librados: setting wanted keys 2023-10-30T12:18:03.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 1 librados: calling monclient init 2023-10-30T12:18:03.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient: init 2023-10-30T12:18:03.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:18:03.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59dc0b7c10 0x7f59dc14c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:03.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f749806a030 0x7f74980703b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient(hunting): picked mon.noname-c con 0x7f749806a030 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f74980ae9b0 0x7f749806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient(hunting): picked mon.noname-a con 0x7f74980ae9b0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:03.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient(hunting): picked mon.noname-a con 0x7f59dc0b7c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:03.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7498141ee0 0x7f7498137660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient(hunting): picked mon.noname-b con 0x7f7498141ee0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:03.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749806a7f0 con 0x7f74980ae9b0 2023-10-30T12:18:03.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7498070fa0 con 0x7f7498141ee0 2023-10-30T12:18:03.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749806ece0 con 0x7f749806a030 2023-10-30T12:18:03.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f749c8e5640 10 monclient(hunting): authenticate will time out at 1575.513702s 2023-10-30T12:18:03.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc141940 0x7f59dc1333d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient(hunting): picked mon.noname-b con 0x7f59dc141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:03.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59dc14a460 0x7f59dc133910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient(hunting): picked mon.noname-c con 0x7f59dc14a460 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc142940 con 0x7f59dc0b7c10 2023-10-30T12:18:03.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc138af0 con 0x7f59dc141940 2023-10-30T12:18:03.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc139ad0 con 0x7f59dc14a460 2023-10-30T12:18:03.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.308+0000 7f59e44a2640 10 monclient(hunting): authenticate will time out at 1575.513823s 2023-10-30T12:18:03.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.309+0000 7f7eb7fff640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2023-10-30T12:18:03.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.309+0000 7f7eb7fff640 1 -- 172.21.15.148:0/3037561332 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7eb80acb50 msgr2=0x7f7eb81364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.309+0000 7f7eb7fff640 1 --2- 172.21.15.148:0/3037561332 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7eb80acb50 0x7f7eb81364a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.309+0000 7f7eb7fff640 1 -- 172.21.15.148:0/3037561332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb806d6c0 msgr2=0x7f7eb813a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.309+0000 7f7eb7fff640 1 --2- 172.21.15.148:0/3037561332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb806d6c0 0x7f7eb813a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.309+0000 7f7eb7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.309+0000 7f7eb7fff640 1 monclient: found mon.noname-b 2023-10-30T12:18:03.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.309+0000 7f7eb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.309+0000 7f7eb7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.309+0000 7f7eb7fff640 1 -- 172.21.15.148:0/3037561332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb81441c0 con 0x7f7eb806dd80 2023-10-30T12:18:03.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.310+0000 7f331d50e640 1 --2- 172.21.15.148:0/1915954374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33180ae890 0x7f33181263b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f331dd0f640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2023-10-30T12:18:03.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f331dd0f640 1 -- 172.21.15.148:0/1915954374 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3318149e10 msgr2=0x7f33181dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f331dd0f640 1 --2- 172.21.15.148:0/1915954374 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3318149e10 0x7f33181dab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f331dd0f640 1 -- 172.21.15.148:0/1915954374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33180ae890 msgr2=0x7f33181263b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f331dd0f640 1 --2- 172.21.15.148:0/1915954374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33180ae890 0x7f33181263b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f331dd0f640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f331dd0f640 1 monclient: found mon.noname-a 2023-10-30T12:18:03.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f331dd0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f331dd0f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:03.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f331dd0f640 1 -- 172.21.15.148:0/1915954374 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3318144f80 con 0x7f331806d610 2023-10-30T12:18:03.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f88f259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f406bc50 0x7f88f41360b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59838/0 (socket says 172.21.15.148:59838) 2023-10-30T12:18:03.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f88f259c640 1 -- 172.21.15.148:0/4211863896 learned_addr learned my addr 172.21.15.148:0/4211863896 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f88f259c640 10 monclient(hunting): get_auth_request con 0x7f88f406bc50 auth_method 0 2023-10-30T12:18:03.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f88f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f88f259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.311+0000 7f88f259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 1 -- 172.21.15.148:0/2026037036 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c008df0 con 0x7fab740c1750 2023-10-30T12:18:03.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:03.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 10 monclient: dump: 2023-10-30T12:18:03.528 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.528 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.529 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.529 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.529 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.529 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.529 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.530 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.530 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.530 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab84461640 5 monclient: authenticate success, global_id 4284 2023-10-30T12:18:03.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab84461640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:03.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 1 -- 172.21.15.148:0/2026037036 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c0306e0 con 0x7fab740c1750 2023-10-30T12:18:03.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab84461640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:03.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab84461640 4 set_mon_vals no callback set 2023-10-30T12:18:03.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7f74977fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f749806a030 0x7f74980703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7f88f1d9b640 1 --2- 172.21.15.148:0/4211863896 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f88f4070550 0x7f88f40ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7f88f2d9d640 1 --2- 172.21.15.148:0/4211863896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88f4140d10 0x7f88f4137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 1 -- 172.21.15.148:0/2026037036 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c030a80 con 0x7fab740c1750 2023-10-30T12:18:03.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:03.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.313+0000 7fab811fb640 10 monclient: dump: 2023-10-30T12:18:03.535 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.535 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.535 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.535 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.535 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.536 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.536 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.536 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.536 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.536 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f74977fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f749806a030 0x7f74980703b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54434/0 (socket says 172.21.15.148:54434) 2023-10-30T12:18:03.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f74977fe640 1 -- 172.21.15.148:0/673158294 learned_addr learned my addr 172.21.15.148:0/673158294 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f74977fe640 10 monclient(hunting): get_auth_request con 0x7f749806a030 auth_method 0 2023-10-30T12:18:03.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f74977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f74977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f74977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7fab84461640 10 monclient: shutdown 2023-10-30T12:18:03.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7fab84461640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7fab84461640 1 -- 172.21.15.148:0/2026037036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab740c1750 msgr2=0x7fab740c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7fab84461640 1 --2- 172.21.15.148:0/2026037036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab740c1750 0x7fab740c7370 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fab74007390 tx=0x7fab6c02fd40 comp rx=0 tx=0).stop 2023-10-30T12:18:03.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f2d9d640 10 monclient(hunting): get_auth_request con 0x7f88f4140d10 auth_method 0 2023-10-30T12:18:03.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7fab84461640 1 -- 172.21.15.148:0/2026037036 shutdown_connections 2023-10-30T12:18:03.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7fab84461640 1 --2- 172.21.15.148:0/2026037036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab740c1750 0x7fab740c7370 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7fab84461640 1 --2- 172.21.15.148:0/2026037036 >> v2:172.21.15.150:3300/0 conn(0x7fab74034da0 0x7fab740c1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7fab84461640 1 -- 172.21.15.148:0/2026037036 >> 172.21.15.148:0/2026037036 conn(0x7fab740061f0 msgr2=0x7fab74006600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7fab84461640 1 -- 172.21.15.148:0/2026037036 shutdown_connections 2023-10-30T12:18:03.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7fab84461640 1 -- 172.21.15.148:0/2026037036 wait complete. 2023-10-30T12:18:03.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f1d9b640 10 monclient(hunting): get_auth_request con 0x7f88f4070550 auth_method 0 2023-10-30T12:18:03.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.314+0000 7f88f1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fab84461640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fab84461640 10 monclient: monmap: 2023-10-30T12:18:03.544 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.544 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:03.544 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:03.545 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:03.545 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.545 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:03.545 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:03.545 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:03.546 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:03.546 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 1 -- 172.21.15.148:0/1127019227 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4031800 con 0x7fccb0039820 2023-10-30T12:18:03.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:03.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 10 monclient: dump: 2023-10-30T12:18:03.547 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.547 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.547 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.548 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.548 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.548 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.548 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.548 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.549 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.549 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 1 -- 172.21.15.148:0/1127019227 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb4031970 con 0x7fccb0039820 2023-10-30T12:18:03.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbed67640 5 monclient: authenticate success, global_id 4250 2023-10-30T12:18:03.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbed67640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:03.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbed67640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:03.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 1 -- 172.21.15.148:0/1127019227 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4031d10 con 0x7fccb0039820 2023-10-30T12:18:03.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbed67640 4 set_mon_vals no callback set 2023-10-30T12:18:03.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:03.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fccbcd63640 10 monclient: dump: 2023-10-30T12:18:03.552 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.552 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.553 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.553 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.553 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.553 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.553 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.554 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.554 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.554 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fab84461640 1 librados: starting msgr at 2023-10-30T12:18:03.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fab84461640 1 librados: starting objecter 2023-10-30T12:18:03.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fab84461640 1 Processor -- start 2023-10-30T12:18:03.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7f88f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7f88f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7f88f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fab84461640 1 -- start start 2023-10-30T12:18:03.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fab84461640 1 librados: setting wanted keys 2023-10-30T12:18:03.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fab84461640 1 librados: calling monclient init 2023-10-30T12:18:03.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.315+0000 7fab84461640 10 monclient: init 2023-10-30T12:18:03.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 10 monclient: shutdown 2023-10-30T12:18:03.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 1 -- 172.21.15.148:0/1127019227 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb0039820 msgr2=0x7fccb0033890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 1 --2- 172.21.15.148:0/1127019227 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb0039820 0x7fccb0033890 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009c50 tx=0x7fccb402b8a0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 1 -- 172.21.15.148:0/1127019227 shutdown_connections 2023-10-30T12:18:03.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 1 --2- 172.21.15.148:0/1127019227 >> v2:172.21.15.148:3300/0 conn(0x7fccb00364b0 0x7fccb0033dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 1 --2- 172.21.15.148:0/1127019227 >> v2:172.21.15.150:3300/0 conn(0x7fccb00c7db0 0x7fccb00c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 1 --2- 172.21.15.148:0/1127019227 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb0039820 0x7fccb0033890 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 1 -- 172.21.15.148:0/1127019227 >> 172.21.15.148:0/1127019227 conn(0x7fccb0007180 msgr2=0x7fccb0037e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7f7eb7fff640 1 --2- 172.21.15.148:0/3037561332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806dd80 0x7f7eb8135f60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0097d0 tx=0x7f7eac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7f7eb57fa640 10 client.?.objecter ms_handle_connect 0x7f7eb806dd80 2023-10-30T12:18:03.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7f7eb57fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:03.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 1 -- 172.21.15.148:0/1127019227 shutdown_connections 2023-10-30T12:18:03.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fccbed67640 1 -- 172.21.15.148:0/1127019227 wait complete. 2023-10-30T12:18:03.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7f9b98f10640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b94142370 0x7f9b941d7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:18:03.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab7402a660 0x7fab740c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 10 monclient(hunting): picked mon.noname-a con 0x7fab7402a660 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:03.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab74034da0 0x7fab7402ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 10 monclient(hunting): picked mon.noname-b con 0x7fab74034da0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:03.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab740c1750 0x7fab740bc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 10 monclient(hunting): picked mon.noname-c con 0x7fab740c1750 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7402ae40 con 0x7fab7402a660 2023-10-30T12:18:03.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7402f420 con 0x7fab74034da0 2023-10-30T12:18:03.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab740c4d20 con 0x7fab740c1750 2023-10-30T12:18:03.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7fab84461640 10 monclient(hunting): authenticate will time out at 1575.521590s 2023-10-30T12:18:03.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.316+0000 7f9b937fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9b9413f060 0x7f9b941d6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f88f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f88f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f88f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7fccbed67640 10 monclient: build_initial_monmap 2023-10-30T12:18:03.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7fccbed67640 10 monclient: monmap: 2023-10-30T12:18:03.570 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:03.570 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:03.571 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:03.571 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:03.571 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:03.571 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:03.571 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:03.572 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:03.572 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:03.572 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7fccbed67640 1 librados: starting msgr at 2023-10-30T12:18:03.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7fccbed67640 1 librados: starting objecter 2023-10-30T12:18:03.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7fccbed67640 1 Processor -- start 2023-10-30T12:18:03.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f9b937fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9b9413f060 0x7f9b941d6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54430/0 (socket says 172.21.15.148:54430) 2023-10-30T12:18:03.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f9b937fe640 1 -- 172.21.15.148:0/4136505740 learned_addr learned my addr 172.21.15.148:0/4136505740 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f9b937fe640 10 monclient(hunting): get_auth_request con 0x7f9b9413f060 auth_method 0 2023-10-30T12:18:03.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f9b937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f9b937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f9b937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7fccbed67640 1 -- start start 2023-10-30T12:18:03.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f331dd0f640 1 --2- 172.21.15.148:0/1915954374 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f331806d610 0x7f33180ab7b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f330800c480 tx=0x7f330800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f330e7fc640 10 client.?.objecter ms_handle_connect 0x7f331806d610 2023-10-30T12:18:03.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7f330e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:03.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7fccbed67640 1 librados: setting wanted keys 2023-10-30T12:18:03.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7fccbed67640 1 librados: calling monclient init 2023-10-30T12:18:03.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.317+0000 7fccbed67640 10 monclient: init 2023-10-30T12:18:03.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.318+0000 7fccbed67640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:03.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.318+0000 7fccbed67640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:18:03.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.318+0000 7fccbed67640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb00364b0 0x7fccb002f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.318+0000 7fccbed67640 10 monclient(hunting): picked mon.noname-a con 0x7fccb00364b0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:03.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.318+0000 7fccbed67640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccb0039820 0x7fccb002f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.318+0000 7fccbed67640 10 monclient(hunting): picked mon.noname-b con 0x7fccb0039820 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:03.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.318+0000 7fccbed67640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb00c7db0 0x7fccb015cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.318+0000 7f88f2d9d640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2023-10-30T12:18:03.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7f88f2d9d640 1 -- 172.21.15.148:0/4211863896 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f88f4070550 msgr2=0x7f88f40ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7f88f2d9d640 1 --2- 172.21.15.148:0/4211863896 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f88f4070550 0x7f88f40ab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7f88f2d9d640 1 -- 172.21.15.148:0/4211863896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f406bc50 msgr2=0x7f88f41360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7f88f2d9d640 1 --2- 172.21.15.148:0/4211863896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f406bc50 0x7f88f41360b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7f88f2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7f88f2d9d640 1 monclient: found mon.noname-b 2023-10-30T12:18:03.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7f88f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7f88f2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7f88f2d9d640 1 -- 172.21.15.148:0/4211863896 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f4137a70 con 0x7f88f4140d10 2023-10-30T12:18:03.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7fccbed67640 10 monclient(hunting): picked mon.noname-c con 0x7fccb00c7db0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:03.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7fccbed67640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7fccbed67640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0000f50 con 0x7fccb00364b0 2023-10-30T12:18:03.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7fccbed67640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7fccbed67640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb002ff00 con 0x7fccb0039820 2023-10-30T12:18:03.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7fccbed67640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:03.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7fccbed67640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb002fbc0 con 0x7fccb00c7db0 2023-10-30T12:18:03.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7fccbed67640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:03.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.319+0000 7fccbed67640 10 monclient(hunting): authenticate will time out at 1575.524759s 2023-10-30T12:18:03.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.321+0000 7f9b98f10640 10 monclient(hunting): get_auth_request con 0x7f9b94142370 auth_method 0 2023-10-30T12:18:03.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.321+0000 7f9b98f10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.321+0000 7f9b98f10640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.321+0000 7f9b98f10640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.321+0000 7f9b937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.321+0000 7f9b937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.321+0000 7f9b937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.322+0000 7f88f2d9d640 1 --2- 172.21.15.148:0/4211863896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88f4140d10 0x7f88f4137e90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f88ec033000 tx=0x7f88ec004a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.322+0000 7f88e2ffd640 10 client.?.objecter ms_handle_connect 0x7f88f4140d10 2023-10-30T12:18:03.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.322+0000 7f88e2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:03.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f74977fe640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2023-10-30T12:18:03.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f74977fe640 1 -- 172.21.15.148:0/673158294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7498141ee0 msgr2=0x7f7498137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f74977fe640 1 --2- 172.21.15.148:0/673158294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7498141ee0 0x7f7498137660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f74977fe640 1 -- 172.21.15.148:0/673158294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f74980ae9b0 msgr2=0x7f749806faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f74977fe640 1 --2- 172.21.15.148:0/673158294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f74980ae9b0 0x7f749806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f74977fe640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f74977fe640 1 monclient: found mon.noname-c 2023-10-30T12:18:03.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f74977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f74977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f74977fe640 1 -- 172.21.15.148:0/673158294 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7498070a80 con 0x7f749806a030 2023-10-30T12:18:03.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f9b98f10640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f9b98f10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.325+0000 7f9b98f10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 1 -- 172.21.15.148:0/3037561332 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac032950 con 0x7f7eb806dd80 2023-10-30T12:18:03.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:03.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 10 monclient: dump: 2023-10-30T12:18:03.593 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.593 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.593 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.593 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.594 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.594 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.594 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.594 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.594 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.595 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 1 -- 172.21.15.148:0/3037561332 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac03f030 con 0x7f7eb806dd80 2023-10-30T12:18:03.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7ebdd47640 5 monclient: authenticate success, global_id 4315 2023-10-30T12:18:03.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 1 -- 172.21.15.148:0/3037561332 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac032d20 con 0x7f7eb806dd80 2023-10-30T12:18:03.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:03.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb57fa640 10 monclient: dump: 2023-10-30T12:18:03.598 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.598 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.598 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.598 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.599 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.599 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.599 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.599 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.599 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.599 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7ebdd47640 10 monclient: _renew_subs 2023-10-30T12:18:03.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7ebdd47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7ebdd47640 1 -- 172.21.15.148:0/3037561332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb80716c0 con 0x7f7eb806dd80 2023-10-30T12:18:03.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7eb6ffd640 4 set_mon_vals no callback set 2023-10-30T12:18:03.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7ebdd47640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7ebdd47640 10 monclient: _renew_subs 2023-10-30T12:18:03.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7ebdd47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7ebdd47640 1 -- 172.21.15.148:0/3037561332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb813b2a0 con 0x7f7eb806dd80 2023-10-30T12:18:03.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.327+0000 7f7ebdd47640 1 librados: init done 2023-10-30T12:18:03.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f7e9effd640 10 client.4315.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:03.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f7e9effd640 15 client.4315.objecter _session_command_op_assign -1 1 2023-10-30T12:18:03.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f7e9effd640 20 client.4315.objecter _get_session osd=-1 returning homeless 2023-10-30T12:18:03.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f7e9effd640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f7e9effd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f7e9effd640 1 -- 172.21.15.148:0/3037561332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7eb813f220 con 0x7f7eb806dd80 2023-10-30T12:18:03.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f9b93fff640 1 --2- 172.21.15.148:0/4136505740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b940a9750 0x7f9b941cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f9b93fff640 10 monclient(hunting): get_auth_request con 0x7f9b940a9750 auth_method 0 2023-10-30T12:18:03.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f9b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f9b93fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.328+0000 7f9b93fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b98f10640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2023-10-30T12:18:03.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b98f10640 1 -- 172.21.15.148:0/4136505740 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9b9413f060 msgr2=0x7f9b941d6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b98f10640 1 --2- 172.21.15.148:0/4136505740 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9b9413f060 0x7f9b941d6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b98f10640 1 -- 172.21.15.148:0/4136505740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b940a9750 msgr2=0x7f9b941cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b98f10640 1 --2- 172.21.15.148:0/4136505740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b940a9750 0x7f9b941cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b98f10640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b98f10640 1 monclient: found mon.noname-a 2023-10-30T12:18:03.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b98f10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b98f10640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:03.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.329+0000 7f9b98f10640 1 -- 172.21.15.148:0/4136505740 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b941ce340 con 0x7f9b94142370 2023-10-30T12:18:03.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.334+0000 7f330e7fc640 1 -- 172.21.15.148:0/1915954374 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080052d0 con 0x7f331806d610 2023-10-30T12:18:03.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:03.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 10 monclient: dump: 2023-10-30T12:18:03.608 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.608 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.608 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.609 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.609 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.609 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.609 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.609 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.609 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.610 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 1 -- 172.21.15.148:0/1915954374 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330800e040 con 0x7f331806d610 2023-10-30T12:18:03.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 1 -- 172.21.15.148:0/1915954374 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308012630 con 0x7f331806d610 2023-10-30T12:18:03.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:03.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330e7fc640 10 monclient: dump: 2023-10-30T12:18:03.612 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.612 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.612 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.612 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.612 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.612 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.613 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.613 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.613 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.613 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f331ed11640 5 monclient: authenticate success, global_id 4293 2023-10-30T12:18:03.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f331ed11640 10 monclient: _renew_subs 2023-10-30T12:18:03.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f330ffff640 4 set_mon_vals no callback set 2023-10-30T12:18:03.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f331ed11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:03.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33181db060 con 0x7f331806d610 2023-10-30T12:18:03.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f331ed11640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f331ed11640 10 monclient: _renew_subs 2023-10-30T12:18:03.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f331ed11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:03.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3318125b50 con 0x7f331806d610 2023-10-30T12:18:03.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f331ed11640 1 librados: init done 2023-10-30T12:18:03.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 1 -- 172.21.15.148:0/4211863896 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec03d600 con 0x7f88f4140d10 2023-10-30T12:18:03.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:03.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 10 monclient: dump: 2023-10-30T12:18:03.617 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.617 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.618 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.618 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.618 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.618 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.618 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.619 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.619 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.619 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 1 -- 172.21.15.148:0/4211863896 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec03d770 con 0x7f88f4140d10 2023-10-30T12:18:03.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 1 -- 172.21.15.148:0/4211863896 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec03db10 con 0x7f88f4140d10 2023-10-30T12:18:03.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:03.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88e2ffd640 10 monclient: dump: 2023-10-30T12:18:03.621 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.621 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.621 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.621 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.622 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.622 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.622 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.623 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.623 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.623 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88f0d99640 4 set_mon_vals no callback set 2023-10-30T12:18:03.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88f359e640 5 monclient: authenticate success, global_id 4318 2023-10-30T12:18:03.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88f359e640 10 monclient: _renew_subs 2023-10-30T12:18:03.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88f359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88f359e640 1 -- 172.21.15.148:0/4211863896 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f40abb20 con 0x7f88f4140d10 2023-10-30T12:18:03.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88f359e640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88f359e640 10 monclient: _renew_subs 2023-10-30T12:18:03.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88f359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88f359e640 1 -- 172.21.15.148:0/4211863896 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f41383d0 con 0x7f88f4140d10 2023-10-30T12:18:03.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.335+0000 7f88f359e640 1 librados: init done 2023-10-30T12:18:03.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f331ed11640 10 client.4293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:03.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f331ed11640 15 client.4293.objecter _session_command_op_assign -1 1 2023-10-30T12:18:03.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f331ed11640 20 client.4293.objecter _get_session osd=-1 returning homeless 2023-10-30T12:18:03.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f331ed11640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f331ed11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:03.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f331806fd40 con 0x7f331806d610 2023-10-30T12:18:03.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f9b90ff9640 10 client.?.objecter ms_handle_connect 0x7f9b94142370 2023-10-30T12:18:03.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f9b90ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:03.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f9b98f10640 1 --2- 172.21.15.148:0/4136505740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b94142370 0x7f9b941d7440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b9406bcb0 tx=0x7f9b880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f88f359e640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:03.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f88f359e640 15 client.4318.objecter _session_command_op_assign -1 1 2023-10-30T12:18:03.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f88f359e640 20 client.4318.objecter _get_session osd=-1 returning homeless 2023-10-30T12:18:03.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f88f359e640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f88f359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f88f359e640 1 -- 172.21.15.148:0/4211863896 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f88f4070c70 con 0x7f88f4140d10 2023-10-30T12:18:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7fab819fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab74034da0 0x7fab7402ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.336+0000 7f59e1a3d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc141940 0x7f59dc1333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.337+0000 7fab819fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab74034da0 0x7fab7402ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48736/0 (socket says 172.21.15.148:48736) 2023-10-30T12:18:03.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.337+0000 7fab819fc640 1 -- 172.21.15.148:0/212674265 learned_addr learned my addr 172.21.15.148:0/212674265 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.337+0000 7fab819fc640 10 monclient(hunting): get_auth_request con 0x7fab74034da0 auth_method 0 2023-10-30T12:18:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.337+0000 7fab819fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.337+0000 7fab819fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.337+0000 7fab819fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.338+0000 7f59e1a3d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc141940 0x7f59dc1333d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48726/0 (socket says 172.21.15.148:48726) 2023-10-30T12:18:03.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.338+0000 7f59e1a3d640 1 -- 172.21.15.148:0/3896420655 learned_addr learned my addr 172.21.15.148:0/3896420655 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.338+0000 7f59e1a3d640 10 monclient(hunting): get_auth_request con 0x7f59dc141940 auth_method 0 2023-10-30T12:18:03.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.338+0000 7f59e1a3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.338+0000 7f59e1a3d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.338+0000 7f59e1a3d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.338+0000 7f59e1a3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.338+0000 7f59e1a3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.338+0000 7f59e1a3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.339+0000 7fab819fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.339+0000 7fab819fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.339+0000 7fab819fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.339+0000 7f74977fe640 1 --2- 172.21.15.148:0/673158294 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f749806a030 0x7f74980703b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f749000c7d0 tx=0x7f749000cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.339+0000 7f747ffff640 10 client.?.objecter ms_handle_connect 0x7f749806a030 2023-10-30T12:18:03.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.339+0000 7f747ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:03.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.340+0000 7fccbe566640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb00c7db0 0x7fccb015cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.340+0000 7fccbe566640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb00c7db0 0x7fccb015cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54466/0 (socket says 172.21.15.148:54466) 2023-10-30T12:18:03.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.340+0000 7fccbe566640 1 -- 172.21.15.148:0/1248287301 learned_addr learned my addr 172.21.15.148:0/1248287301 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:03.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.340+0000 7fccbe566640 10 monclient(hunting): get_auth_request con 0x7fccb00c7db0 auth_method 0 2023-10-30T12:18:03.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.340+0000 7fccbe566640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.340+0000 7fccbe566640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.340+0000 7fccbe566640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.341+0000 7f59e2a3f640 1 --2- 172.21.15.148:0/3896420655 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59dc14a460 0x7f59dc133910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.341+0000 7fab829fe640 1 --2- 172.21.15.148:0/212674265 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab740c1750 0x7fab740bc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.342+0000 7f59e1a3d640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2023-10-30T12:18:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.342+0000 7f59e1a3d640 1 -- 172.21.15.148:0/3896420655 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59dc14a460 msgr2=0x7f59dc133910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.342+0000 7f59e1a3d640 1 --2- 172.21.15.148:0/3896420655 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59dc14a460 0x7f59dc133910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.342+0000 7f59e1a3d640 1 -- 172.21.15.148:0/3896420655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59dc0b7c10 msgr2=0x7f59dc14c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.342+0000 7f59e1a3d640 1 --2- 172.21.15.148:0/3896420655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59dc0b7c10 0x7f59dc14c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.342+0000 7f59e1a3d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.342+0000 7f59e1a3d640 1 monclient: found mon.noname-b 2023-10-30T12:18:03.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.342+0000 7f59e1a3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.342+0000 7f59e1a3d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.342+0000 7f59e1a3d640 1 -- 172.21.15.148:0/3896420655 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59dc139db0 con 0x7f59dc141940 2023-10-30T12:18:03.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.343+0000 7fab829fe640 10 monclient(hunting): get_auth_request con 0x7fab740c1750 auth_method 0 2023-10-30T12:18:03.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.343+0000 7fab829fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:03.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.343+0000 7fab829fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:03.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.343+0000 7fab829fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:03.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.344+0000 7fab829fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.344+0000 7fab829fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.344+0000 7fab829fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fab819fc640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2023-10-30T12:18:03.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fab819fc640 1 -- 172.21.15.148:0/212674265 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab740c1750 msgr2=0x7fab740bc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fab819fc640 1 --2- 172.21.15.148:0/212674265 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab740c1750 0x7fab740bc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fab819fc640 1 -- 172.21.15.148:0/212674265 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab7402a660 msgr2=0x7fab740c40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fab819fc640 1 --2- 172.21.15.148:0/212674265 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab7402a660 0x7fab740c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fab819fc640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fab819fc640 1 monclient: found mon.noname-b 2023-10-30T12:18:03.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fab819fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fab819fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fab819fc640 1 -- 172.21.15.148:0/212674265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7402e500 con 0x7fab74034da0 2023-10-30T12:18:03.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fccbe566640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:03.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fccbe566640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:03.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.346+0000 7fccbe566640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:03.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 1 -- 172.21.15.148:0/673158294 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490003d80 con 0x7f749806a030 2023-10-30T12:18:03.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:03.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 10 monclient: dump: 2023-10-30T12:18:03.649 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.649 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.649 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.650 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.650 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.650 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.650 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.650 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.651 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.651 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 1 -- 172.21.15.148:0/673158294 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749000e040 con 0x7f749806a030 2023-10-30T12:18:03.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f749c8e5640 5 monclient: authenticate success, global_id 4256 2023-10-30T12:18:03.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 1 -- 172.21.15.148:0/673158294 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74900052b0 con 0x7f749806a030 2023-10-30T12:18:03.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:03.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f747ffff640 10 monclient: dump: 2023-10-30T12:18:03.653 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.653 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.653 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.653 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.654 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.654 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.654 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.654 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.654 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.654 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f749c8e5640 10 monclient: _renew_subs 2023-10-30T12:18:03.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f749c8e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74980700f0 con 0x7f749806a030 2023-10-30T12:18:03.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f7495ffb640 4 set_mon_vals no callback set 2023-10-30T12:18:03.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f749c8e5640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f749c8e5640 10 monclient: _renew_subs 2023-10-30T12:18:03.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f749c8e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7498137de0 con 0x7f749806a030 2023-10-30T12:18:03.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f749c8e5640 1 librados: init done 2023-10-30T12:18:03.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f59e1a3d640 1 --2- 172.21.15.148:0/3896420655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc141940 0x7f59dc1333d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00c410 tx=0x7f59cc00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f59d2ffd640 10 client.?.objecter ms_handle_connect 0x7f59dc141940 2023-10-30T12:18:03.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.347+0000 7f59d2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:03.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.348+0000 7f749c8e5640 10 client.4256.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:03.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.348+0000 7f749c8e5640 15 client.4256.objecter _session_command_op_assign -1 1 2023-10-30T12:18:03.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.348+0000 7f749c8e5640 20 client.4256.objecter _get_session osd=-1 returning homeless 2023-10-30T12:18:03.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.348+0000 7f749c8e5640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.348+0000 7f749c8e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.348+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74980aedf0 con 0x7f749806a030 2023-10-30T12:18:03.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.349+0000 7fccbd564640 1 --2- 172.21.15.148:0/1248287301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccb0039820 0x7fccb002f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.350+0000 7fccbe566640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2023-10-30T12:18:03.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.350+0000 7fccbe566640 1 -- 172.21.15.148:0/1248287301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccb0039820 msgr2=0x7fccb002f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.350+0000 7fccbe566640 1 --2- 172.21.15.148:0/1248287301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccb0039820 0x7fccb002f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.350+0000 7fccbe566640 1 -- 172.21.15.148:0/1248287301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb00364b0 msgr2=0x7fccb002f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.350+0000 7fccbe566640 1 --2- 172.21.15.148:0/1248287301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb00364b0 0x7fccb002f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.350+0000 7fccbe566640 10 monclient: _finish_hunting 0 2023-10-30T12:18:03.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.350+0000 7fccbe566640 1 monclient: found mon.noname-c 2023-10-30T12:18:03.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.350+0000 7fccbe566640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:03.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.350+0000 7fccbe566640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.350+0000 7fccbe566640 1 -- 172.21.15.148:0/1248287301 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0030390 con 0x7fccb00c7db0 2023-10-30T12:18:03.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.357+0000 7fccbe566640 1 --2- 172.21.15.148:0/1248287301 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb00c7db0 0x7fccb015cdf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fccb80726d0 tx=0x7fccb8067a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.357+0000 7fccae7fc640 10 client.?.objecter ms_handle_connect 0x7fccb00c7db0 2023-10-30T12:18:03.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.357+0000 7fccae7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:03.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.362+0000 7fab819fc640 1 --2- 172.21.15.148:0/212674265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab74034da0 0x7fab7402ed70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab6c03a040 tx=0x7fab6c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.362+0000 7fab72ffd640 10 client.?.objecter ms_handle_connect 0x7fab74034da0 2023-10-30T12:18:03.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.362+0000 7fab72ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:03.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.369+0000 7f330e7fc640 1 -- 172.21.15.148:0/1915954374 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f33080127a0 con 0x7f331806d610 2023-10-30T12:18:03.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.370+0000 7f330e7fc640 1 --2- 172.21.15.148:0/1915954374 >> v2:172.21.15.148:6800/3205090179 conn(0x7f32fc06c810 0x7f32fc06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.370+0000 7f331d50e640 1 --2- 172.21.15.148:0/1915954374 >> v2:172.21.15.148:6800/3205090179 conn(0x7f32fc06c810 0x7f32fc06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.370+0000 7f331d50e640 10 monclient: get_auth_request con 0x7f32fc06c810 auth_method 0 2023-10-30T12:18:03.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.370+0000 7f330e7fc640 10 client.4293.objecter ms_handle_connect 0x7f32fc06c810 2023-10-30T12:18:03.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.370+0000 7f331d50e640 1 --2- 172.21.15.148:0/1915954374 >> v2:172.21.15.148:6800/3205090179 conn(0x7f32fc06c810 0x7f32fc06ecd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3314036000 tx=0x7f331403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 1 -- 172.21.15.148:0/1915954374 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f3308058960 con 0x7f331806d610 2023-10-30T12:18:03.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 10 client.4293.objecter ms_dispatch 0x7f3318000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:03.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 3 client.4293.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:03.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 3 client.4293.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:03.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 10 client.4293.objecter checking command 1 2023-10-30T12:18:03.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 1 --2- 172.21.15.148:0/1915954374 >> v2:172.21.15.148:6805/2395694124 conn(0x7f32fc072620 0x7f32fc074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 20 client.4293.objecter _get_session s=0x7f3308012630 osd=0 3 2023-10-30T12:18:03.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 20 client.4293.objecter put_session s=0x7f3308012630 osd=0 3 2023-10-30T12:18:03.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 15 client.4293.objecter _session_command_op_remove -1 1 2023-10-30T12:18:03.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 20 client.4293.objecter _get_session s=0x7f3308012630 osd=0 3 2023-10-30T12:18:03.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 20 client.4293.objecter get_session s=0x7f3308012630 osd=0 3 2023-10-30T12:18:03.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 15 client.4293.objecter _session_command_op_assign 0 1 2023-10-30T12:18:03.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 20 client.4293.objecter put_session s=0x7f3308012630 osd=0 4 2023-10-30T12:18:03.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 10 client.4293.objecter _send_command 1 2023-10-30T12:18:03.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 1 -- 172.21.15.148:0/1915954374 --> v2:172.21.15.148:6805/2395694124 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33080052d0 con 0x7f32fc072620 2023-10-30T12:18:03.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 20 client.4293.objecter dump_active .. 0 homeless 2023-10-30T12:18:03.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 1 -- 172.21.15.148:0/1915954374 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33080543d0 con 0x7f331806d610 2023-10-30T12:18:03.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.378+0000 7f330e7fc640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-10-30T12:18:03.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.379+0000 7f331e510640 1 --2- 172.21.15.148:0/1915954374 >> v2:172.21.15.148:6805/2395694124 conn(0x7f32fc072620 0x7f32fc074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.379+0000 7f7eb57fa640 1 -- 172.21.15.148:0/3037561332 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f7eac04e440 con 0x7f7eb806dd80 2023-10-30T12:18:03.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.379+0000 7f331e510640 10 monclient: get_auth_request con 0x7f32fc072620 auth_method 0 2023-10-30T12:18:03.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.379+0000 7f7eb57fa640 1 --2- 172.21.15.148:0/3037561332 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7ea406c7f0 0x7f7ea406ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.379+0000 7f7ebcd45640 1 --2- 172.21.15.148:0/3037561332 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7ea406c7f0 0x7f7ea406ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.379+0000 7f7ebcd45640 10 monclient: get_auth_request con 0x7f7ea406c7f0 auth_method 0 2023-10-30T12:18:03.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.380+0000 7f7eb57fa640 10 client.4315.objecter ms_handle_connect 0x7f7ea406c7f0 2023-10-30T12:18:03.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.380+0000 7f7ebcd45640 1 --2- 172.21.15.148:0/3037561332 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7ea406c7f0 0x7f7ea406ecb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0002070 tx=0x7f7eb0001660 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.383+0000 7f330e7fc640 10 client.4293.objecter ms_handle_connect 0x7f32fc072620 2023-10-30T12:18:03.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.383+0000 7f331e510640 1 --2- 172.21.15.148:0/1915954374 >> v2:172.21.15.148:6805/2395694124 conn(0x7f32fc072620 0x7f32fc074a40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.384+0000 7f9b90ff9640 1 -- 172.21.15.148:0/4136505740 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88032880 con 0x7f9b94142370 2023-10-30T12:18:03.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.384+0000 7f9b90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:03.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 10 monclient: dump: 2023-10-30T12:18:03.676 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.676 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.676 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.677 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.677 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.677 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.677 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.677 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.677 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.678 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 1 -- 172.21.15.148:0/4136505740 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8803f030 con 0x7f9b94142370 2023-10-30T12:18:03.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 1 -- 172.21.15.148:0/4136505740 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88032c50 con 0x7f9b94142370 2023-10-30T12:18:03.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:03.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b90ff9640 10 monclient: dump: 2023-10-30T12:18:03.680 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.680 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.680 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.680 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.680 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.681 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.681 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.681 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.681 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.681 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b927fc640 4 set_mon_vals no callback set 2023-10-30T12:18:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b9a973640 5 monclient: authenticate success, global_id 4299 2023-10-30T12:18:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b9a973640 10 monclient: _renew_subs 2023-10-30T12:18:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b9a973640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b941d7980 con 0x7f9b94142370 2023-10-30T12:18:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b9a973640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b9a973640 10 monclient: _renew_subs 2023-10-30T12:18:03.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b9a973640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:03.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b94139c60 con 0x7f9b94142370 2023-10-30T12:18:03.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.385+0000 7f9b9a973640 1 librados: init done 2023-10-30T12:18:03.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.386+0000 7f9b9a973640 10 client.4299.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:03.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.386+0000 7f9b9a973640 15 client.4299.objecter _session_command_op_assign -1 1 2023-10-30T12:18:03.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.386+0000 7f9b9a973640 20 client.4299.objecter _get_session osd=-1 returning homeless 2023-10-30T12:18:03.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.386+0000 7f9b9a973640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.386+0000 7f9b9a973640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:03.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.386+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9b94071370 con 0x7f9b94142370 2023-10-30T12:18:03.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccae7fc640 1 -- 172.21.15.148:0/1248287301 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb804e6d0 con 0x7fccb00c7db0 2023-10-30T12:18:03.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:03.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccae7fc640 10 monclient: dump: 2023-10-30T12:18:03.686 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.686 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.686 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.687 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.687 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.687 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.687 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.687 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.688 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.688 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccae7fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccae7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccbed67640 5 monclient: authenticate success, global_id 4268 2023-10-30T12:18:03.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccbed67640 10 monclient: _renew_subs 2023-10-30T12:18:03.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccbed67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb00cb5e0 con 0x7fccb00c7db0 2023-10-30T12:18:03.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccbed67640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccbed67640 10 monclient: _renew_subs 2023-10-30T12:18:03.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccbed67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb015d510 con 0x7fccb00c7db0 2023-10-30T12:18:03.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccbed67640 1 librados: init done 2023-10-30T12:18:03.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7f747ffff640 1 -- 172.21.15.148:0/673158294 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f7490005420 con 0x7f749806a030 2023-10-30T12:18:03.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7f747ffff640 1 --2- 172.21.15.148:0/673158294 >> v2:172.21.15.148:6800/3205090179 conn(0x7f748406c810 0x7f748406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.387+0000 7fccbed67640 10 client.4268.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:03.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.388+0000 7fccbed67640 15 client.4268.objecter _session_command_op_assign -1 1 2023-10-30T12:18:03.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.388+0000 7fccbed67640 20 client.4268.objecter _get_session osd=-1 returning homeless 2023-10-30T12:18:03.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.388+0000 7fccbed67640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.388+0000 7fccbed67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:03.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.388+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fccb0037750 con 0x7fccb00c7db0 2023-10-30T12:18:03.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.388+0000 7f7496ffd640 1 --2- 172.21.15.148:0/673158294 >> v2:172.21.15.148:6800/3205090179 conn(0x7f748406c810 0x7f748406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.388+0000 7f7496ffd640 10 monclient: get_auth_request con 0x7f748406c810 auth_method 0 2023-10-30T12:18:03.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.388+0000 7f747ffff640 10 client.4256.objecter ms_handle_connect 0x7f748406c810 2023-10-30T12:18:03.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.388+0000 7f7496ffd640 1 --2- 172.21.15.148:0/673158294 >> v2:172.21.15.148:6800/3205090179 conn(0x7f748406c810 0x7f748406ecd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f748c036000 tx=0x7f748c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 1 -- 172.21.15.148:0/673158294 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f74900584b0 con 0x7f749806a030 2023-10-30T12:18:03.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 10 client.4256.objecter ms_dispatch 0x7f7498000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:03.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 3 client.4256.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:03.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 3 client.4256.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:03.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 10 client.4256.objecter checking command 1 2023-10-30T12:18:03.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 1 --2- 172.21.15.148:0/673158294 >> v2:172.21.15.148:6806/2233349848 conn(0x7f7484072620 0x7f7484074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 20 client.4256.objecter _get_session s=0x7f74900052b0 osd=3 3 2023-10-30T12:18:03.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 20 client.4256.objecter put_session s=0x7f74900052b0 osd=3 3 2023-10-30T12:18:03.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 15 client.4256.objecter _session_command_op_remove -1 1 2023-10-30T12:18:03.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 20 client.4256.objecter _get_session s=0x7f74900052b0 osd=3 3 2023-10-30T12:18:03.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 20 client.4256.objecter get_session s=0x7f74900052b0 osd=3 3 2023-10-30T12:18:03.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 15 client.4256.objecter _session_command_op_assign 3 1 2023-10-30T12:18:03.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 20 client.4256.objecter put_session s=0x7f74900052b0 osd=3 4 2023-10-30T12:18:03.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 10 client.4256.objecter _send_command 1 2023-10-30T12:18:03.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 1 -- 172.21.15.148:0/673158294 --> v2:172.21.15.148:6806/2233349848 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7490003d80 con 0x7f7484072620 2023-10-30T12:18:03.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 20 client.4256.objecter dump_active .. 0 homeless 2023-10-30T12:18:03.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 1 -- 172.21.15.148:0/673158294 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7490001830 con 0x7f749806a030 2023-10-30T12:18:03.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f747ffff640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-10-30T12:18:03.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.392+0000 7f7497fff640 1 --2- 172.21.15.148:0/673158294 >> v2:172.21.15.148:6806/2233349848 conn(0x7f7484072620 0x7f7484074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.393+0000 7f7497fff640 10 monclient: get_auth_request con 0x7f7484072620 auth_method 0 2023-10-30T12:18:03.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 1 -- 172.21.15.148:0/3037561332 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f7eac02f050 con 0x7f7eb806dd80 2023-10-30T12:18:03.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 10 client.4315.objecter ms_dispatch 0x7f7eb8000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:03.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 3 client.4315.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:03.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 3 client.4315.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:03.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 10 client.4315.objecter checking command 1 2023-10-30T12:18:03.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 1 --2- 172.21.15.148:0/3037561332 >> v2:172.21.15.148:6801/2368806934 conn(0x7f7ea4072600 0x7f7ea4074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 20 client.4315.objecter _get_session s=0x7f7eac032d20 osd=2 3 2023-10-30T12:18:03.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 20 client.4315.objecter put_session s=0x7f7eac032d20 osd=2 3 2023-10-30T12:18:03.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 15 client.4315.objecter _session_command_op_remove -1 1 2023-10-30T12:18:03.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 20 client.4315.objecter _get_session s=0x7f7eac032d20 osd=2 3 2023-10-30T12:18:03.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 20 client.4315.objecter get_session s=0x7f7eac032d20 osd=2 3 2023-10-30T12:18:03.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 15 client.4315.objecter _session_command_op_assign 2 1 2023-10-30T12:18:03.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 20 client.4315.objecter put_session s=0x7f7eac032d20 osd=2 4 2023-10-30T12:18:03.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 10 client.4315.objecter _send_command 1 2023-10-30T12:18:03.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 1 -- 172.21.15.148:0/3037561332 --> v2:172.21.15.148:6801/2368806934 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7eac032950 con 0x7f7ea4072600 2023-10-30T12:18:03.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 20 client.4315.objecter dump_active .. 0 homeless 2023-10-30T12:18:03.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 1 -- 172.21.15.148:0/3037561332 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7eac032e90 con 0x7f7eb806dd80 2023-10-30T12:18:03.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.394+0000 7f7eb57fa640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-10-30T12:18:03.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.395+0000 7f7ebd546640 1 --2- 172.21.15.148:0/3037561332 >> v2:172.21.15.148:6801/2368806934 conn(0x7f7ea4072600 0x7f7ea4074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.395+0000 7f7ebd546640 10 monclient: get_auth_request con 0x7f7ea4072600 auth_method 0 2023-10-30T12:18:03.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.395+0000 7fccae7fc640 1 -- 172.21.15.148:0/1248287301 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb0037750 con 0x7fccb00c7db0 2023-10-30T12:18:03.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.395+0000 7fccae7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.395+0000 7fccae7fc640 1 -- 172.21.15.148:0/1248287301 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb80721c0 con 0x7fccb00c7db0 2023-10-30T12:18:03.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.395+0000 7fccae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.395+0000 7fccae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:03.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.395+0000 7fccae7fc640 10 monclient: dump: 2023-10-30T12:18:03.707 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.707 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.707 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.708 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.708 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.708 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.708 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.708 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.709 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.709 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.395+0000 7fccaffff640 4 set_mon_vals no callback set 2023-10-30T12:18:03.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.396+0000 7f7497fff640 1 --2- 172.21.15.148:0/673158294 >> v2:172.21.15.148:6806/2233349848 conn(0x7f7484072620 0x7f7484074a40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.396+0000 7f747ffff640 10 client.4256.objecter ms_handle_connect 0x7f7484072620 2023-10-30T12:18:03.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 1 -- 172.21.15.148:0/212674265 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c00a450 con 0x7fab74034da0 2023-10-30T12:18:03.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:03.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 10 monclient: dump: 2023-10-30T12:18:03.712 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.712 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.712 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.712 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.712 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.713 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.713 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.713 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.713 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.713 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 1 -- 172.21.15.148:0/212674265 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c00a5c0 con 0x7fab74034da0 2023-10-30T12:18:03.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 1 -- 172.21.15.148:0/212674265 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c042410 con 0x7fab74034da0 2023-10-30T12:18:03.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:03.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab72ffd640 10 monclient: dump: 2023-10-30T12:18:03.715 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.715 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.716 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.716 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.716 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.716 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.716 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.716 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.717 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.717 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab84461640 5 monclient: authenticate success, global_id 4327 2023-10-30T12:18:03.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab84461640 10 monclient: _renew_subs 2023-10-30T12:18:03.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab84461640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab740bc920 con 0x7fab74034da0 2023-10-30T12:18:03.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab84461640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab84461640 10 monclient: _renew_subs 2023-10-30T12:18:03.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab84461640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab740bccd0 con 0x7fab74034da0 2023-10-30T12:18:03.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.398+0000 7fab84461640 1 librados: init done 2023-10-30T12:18:03.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.399+0000 7fab84461640 10 client.4327.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:03.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.399+0000 7fab84461640 15 client.4327.objecter _session_command_op_assign -1 1 2023-10-30T12:18:03.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.399+0000 7fab84461640 20 client.4327.objecter _get_session osd=-1 returning homeless 2023-10-30T12:18:03.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.399+0000 7fab84461640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.399+0000 7fab84461640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.399+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fab74035180 con 0x7fab74034da0 2023-10-30T12:18:03.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.399+0000 7f59d2ffd640 1 -- 172.21.15.148:0/3896420655 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc0035f0 con 0x7f59dc141940 2023-10-30T12:18:03.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:03.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 10 monclient: dump: 2023-10-30T12:18:03.722 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.722 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.722 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.723 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.723 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.723 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.723 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.723 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.724 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.724 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:03.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:03.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:03.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 1 -- 172.21.15.148:0/3896420655 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59cc00e040 con 0x7f59dc141940 2023-10-30T12:18:03.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:03.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 1 -- 172.21.15.148:0/3896420655 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc0039e0 con 0x7f59dc141940 2023-10-30T12:18:03.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:03.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:03.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59d2ffd640 10 monclient: dump: 2023-10-30T12:18:03.726 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:03.727 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:03.727 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.727 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:03.727 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:03.727 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:03.728 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:03.728 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:03.728 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:03.728 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:03.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59e44a2640 5 monclient: authenticate success, global_id 4324 2023-10-30T12:18:03.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59e44a2640 10 monclient: _renew_subs 2023-10-30T12:18:03.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59e44a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59e44a2640 1 -- 172.21.15.148:0/3896420655 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59dc133e50 con 0x7f59dc141940 2023-10-30T12:18:03.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59e44a2640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59e44a2640 10 monclient: _renew_subs 2023-10-30T12:18:03.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59e44a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59e44a2640 1 -- 172.21.15.148:0/3896420655 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59dc14b770 con 0x7f59dc141940 2023-10-30T12:18:03.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.400+0000 7f59e44a2640 1 librados: init done 2023-10-30T12:18:03.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.401+0000 7f59d0ff9640 10 client.4324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:03.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.401+0000 7f59d0ff9640 15 client.4324.objecter _session_command_op_assign -1 1 2023-10-30T12:18:03.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.401+0000 7f59d0ff9640 20 client.4324.objecter _get_session osd=-1 returning homeless 2023-10-30T12:18:03.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.401+0000 7f59d0ff9640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:03.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.401+0000 7f59d0ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:03.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.401+0000 7f59d0ff9640 1 -- 172.21.15.148:0/3896420655 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f59c40028a0 con 0x7f59dc141940 2023-10-30T12:18:03.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.401+0000 7fab809fa640 4 set_mon_vals no callback set 2023-10-30T12:18:03.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.402+0000 7f59e0a3b640 4 set_mon_vals no callback set 2023-10-30T12:18:03.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.403+0000 7f7eb57fa640 10 client.4315.objecter ms_handle_connect 0x7f7ea4072600 2023-10-30T12:18:03.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.403+0000 7f7ebd546640 1 --2- 172.21.15.148:0/3037561332 >> v2:172.21.15.148:6801/2368806934 conn(0x7f7ea4072600 0x7f7ea4074a20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.412+0000 7f88e2ffd640 1 -- 172.21.15.148:0/4211863896 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f88ec03b040 con 0x7f88f4140d10 2023-10-30T12:18:03.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.412+0000 7f88e2ffd640 1 --2- 172.21.15.148:0/4211863896 >> v2:172.21.15.148:6800/3205090179 conn(0x7f88d406c810 0x7f88d406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.413+0000 7f88f259c640 1 --2- 172.21.15.148:0/4211863896 >> v2:172.21.15.148:6800/3205090179 conn(0x7f88d406c810 0x7f88d406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.413+0000 7f88f259c640 10 monclient: get_auth_request con 0x7f88d406c810 auth_method 0 2023-10-30T12:18:03.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.413+0000 7f88e2ffd640 10 client.4318.objecter ms_handle_connect 0x7f88d406c810 2023-10-30T12:18:03.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.413+0000 7f88f259c640 1 --2- 172.21.15.148:0/4211863896 >> v2:172.21.15.148:6800/3205090179 conn(0x7f88d406c810 0x7f88d406ecd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009a40 tx=0x7f88e80091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.414+0000 7f9b90ff9640 1 -- 172.21.15.148:0/4136505740 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f9b880453f0 con 0x7f9b94142370 2023-10-30T12:18:03.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f9b90ff9640 1 --2- 172.21.15.148:0/4136505740 >> v2:172.21.15.148:6800/3205090179 conn(0x7f9b8006c7c0 0x7f9b8006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 1 -- 172.21.15.148:0/4211863896 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f88ec07a540 con 0x7f88f4140d10 2023-10-30T12:18:03.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 10 client.4318.objecter ms_dispatch 0x7f88f4000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:03.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 3 client.4318.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:03.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 3 client.4318.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:03.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 10 client.4318.objecter checking command 1 2023-10-30T12:18:03.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 1 --2- 172.21.15.148:0/4211863896 >> v2:172.21.15.150:6812/3463178370 conn(0x7f88d4072620 0x7f88d4074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 20 client.4318.objecter _get_session s=0x7f88ec03db10 osd=7 3 2023-10-30T12:18:03.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 20 client.4318.objecter put_session s=0x7f88ec03db10 osd=7 3 2023-10-30T12:18:03.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 15 client.4318.objecter _session_command_op_remove -1 1 2023-10-30T12:18:03.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 20 client.4318.objecter _get_session s=0x7f88ec03db10 osd=7 3 2023-10-30T12:18:03.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 20 client.4318.objecter get_session s=0x7f88ec03db10 osd=7 3 2023-10-30T12:18:03.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 15 client.4318.objecter _session_command_op_assign 7 1 2023-10-30T12:18:03.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 20 client.4318.objecter put_session s=0x7f88ec03db10 osd=7 4 2023-10-30T12:18:03.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 10 client.4318.objecter _send_command 1 2023-10-30T12:18:03.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 1 -- 172.21.15.148:0/4211863896 --> v2:172.21.15.150:6812/3463178370 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88ec03d600 con 0x7f88d4072620 2023-10-30T12:18:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f88e2ffd640 20 client.4318.objecter dump_active .. 0 homeless 2023-10-30T12:18:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f9b93fff640 1 --2- 172.21.15.148:0/4136505740 >> v2:172.21.15.148:6800/3205090179 conn(0x7f9b8006c7c0 0x7f9b8006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.415+0000 7f9b93fff640 10 monclient: get_auth_request con 0x7f9b8006c7c0 auth_method 0 2023-10-30T12:18:03.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.416+0000 7f9b93fff640 1 --2- 172.21.15.148:0/4136505740 >> v2:172.21.15.148:6800/3205090179 conn(0x7f9b8006c7c0 0x7f9b8006ec80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009a70 tx=0x7f9b84009110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.416+0000 7f9b90ff9640 10 client.4299.objecter ms_handle_connect 0x7f9b8006c7c0 2023-10-30T12:18:03.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f88f1d9b640 1 --2- 172.21.15.148:0/4211863896 >> v2:172.21.15.150:6812/3463178370 conn(0x7f88d4072620 0x7f88d4074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 1 -- 172.21.15.148:0/4136505740 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f9b88056050 con 0x7f9b94142370 2023-10-30T12:18:03.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 10 client.4299.objecter ms_dispatch 0x7f9b94000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:03.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 3 client.4299.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:03.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 3 client.4299.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:03.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f88f1d9b640 10 monclient: get_auth_request con 0x7f88d4072620 auth_method 0 2023-10-30T12:18:03.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 10 client.4299.objecter checking command 1 2023-10-30T12:18:03.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 1 --2- 172.21.15.148:0/4136505740 >> v2:172.21.15.150:6800/2604225934 conn(0x7f9b80072f10 0x7f9b800753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 20 client.4299.objecter _get_session s=0x7f9b88032c50 osd=6 3 2023-10-30T12:18:03.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 20 client.4299.objecter put_session s=0x7f9b88032c50 osd=6 3 2023-10-30T12:18:03.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 15 client.4299.objecter _session_command_op_remove -1 1 2023-10-30T12:18:03.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 20 client.4299.objecter _get_session s=0x7f9b88032c50 osd=6 3 2023-10-30T12:18:03.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 20 client.4299.objecter get_session s=0x7f9b88032c50 osd=6 3 2023-10-30T12:18:03.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 15 client.4299.objecter _session_command_op_assign 6 1 2023-10-30T12:18:03.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 20 client.4299.objecter put_session s=0x7f9b88032c50 osd=6 4 2023-10-30T12:18:03.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 10 client.4299.objecter _send_command 1 2023-10-30T12:18:03.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 1 -- 172.21.15.148:0/4136505740 --> v2:172.21.15.150:6800/2604225934 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b88032880 con 0x7f9b80072f10 2023-10-30T12:18:03.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 20 client.4299.objecter dump_active .. 0 homeless 2023-10-30T12:18:03.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 1 -- 172.21.15.148:0/4136505740 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b880094d0 con 0x7f9b94142370 2023-10-30T12:18:03.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.418+0000 7f9b90ff9640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-10-30T12:18:03.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.419+0000 7f9b937fe640 1 --2- 172.21.15.148:0/4136505740 >> v2:172.21.15.150:6800/2604225934 conn(0x7f9b80072f10 0x7f9b800753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.419+0000 7f9b937fe640 10 monclient: get_auth_request con 0x7f9b80072f10 auth_method 0 2023-10-30T12:18:03.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.423+0000 7f88e2ffd640 1 -- 172.21.15.148:0/4211863896 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88ec009a70 con 0x7f88f4140d10 2023-10-30T12:18:03.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.423+0000 7f88e2ffd640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-10-30T12:18:03.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.423+0000 7f88e2ffd640 10 client.4318.objecter ms_handle_connect 0x7f88d4072620 2023-10-30T12:18:03.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.423+0000 7f88f1d9b640 1 --2- 172.21.15.148:0/4211863896 >> v2:172.21.15.150:6812/3463178370 conn(0x7f88d4072620 0x7f88d4074a40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.424+0000 7f9b937fe640 1 --2- 172.21.15.148:0/4136505740 >> v2:172.21.15.150:6800/2604225934 conn(0x7f9b80072f10 0x7f9b800753b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.424+0000 7f9b90ff9640 10 client.4299.objecter ms_handle_connect 0x7f9b80072f10 2023-10-30T12:18:03.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.432+0000 7fccae7fc640 1 -- 172.21.15.148:0/1248287301 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fccb8093420 con 0x7fccb00c7db0 2023-10-30T12:18:03.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.432+0000 7fccae7fc640 1 --2- 172.21.15.148:0/1248287301 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcc9c06c7c0 0x7fcc9c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.433+0000 7fccbdd65640 1 --2- 172.21.15.148:0/1248287301 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcc9c06c7c0 0x7fcc9c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.433+0000 7fccbdd65640 10 monclient: get_auth_request con 0x7fcc9c06c7c0 auth_method 0 2023-10-30T12:18:03.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.433+0000 7fccbdd65640 1 --2- 172.21.15.148:0/1248287301 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcc9c06c7c0 0x7fcc9c06ec80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fccb4036000 tx=0x7fccb4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.433+0000 7fccae7fc640 10 client.4268.objecter ms_handle_connect 0x7fcc9c06c7c0 2023-10-30T12:18:03.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.436+0000 7fccae7fc640 1 -- 172.21.15.148:0/1248287301 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fccb8072410 con 0x7fccb00c7db0 2023-10-30T12:18:03.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.436+0000 7fccae7fc640 10 client.4268.objecter ms_dispatch 0x7fccb8000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:03.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.436+0000 7fccae7fc640 3 client.4268.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:03.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.436+0000 7fccae7fc640 3 client.4268.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:03.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fab72ffd640 1 -- 172.21.15.148:0/212674265 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fab6c042580 con 0x7fab74034da0 2023-10-30T12:18:03.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 10 client.4268.objecter checking command 1 2023-10-30T12:18:03.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 1 --2- 172.21.15.148:0/1248287301 >> v2:172.21.15.150:6804/4190376848 conn(0x7fcc9c0725d0 0x7fcc9c0749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 20 client.4268.objecter _get_session s=0x7fccb80721c0 osd=5 3 2023-10-30T12:18:03.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 20 client.4268.objecter put_session s=0x7fccb80721c0 osd=5 3 2023-10-30T12:18:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 15 client.4268.objecter _session_command_op_remove -1 1 2023-10-30T12:18:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 20 client.4268.objecter _get_session s=0x7fccb80721c0 osd=5 3 2023-10-30T12:18:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 20 client.4268.objecter get_session s=0x7fccb80721c0 osd=5 3 2023-10-30T12:18:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 15 client.4268.objecter _session_command_op_assign 5 1 2023-10-30T12:18:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 20 client.4268.objecter put_session s=0x7fccb80721c0 osd=5 4 2023-10-30T12:18:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 10 client.4268.objecter _send_command 1 2023-10-30T12:18:03.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 1 -- 172.21.15.148:0/1248287301 --> v2:172.21.15.150:6804/4190376848 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fccb804e6d0 con 0x7fcc9c0725d0 2023-10-30T12:18:03.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 20 client.4268.objecter dump_active .. 0 homeless 2023-10-30T12:18:03.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 1 -- 172.21.15.148:0/1248287301 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccb808e070 con 0x7fccb00c7db0 2023-10-30T12:18:03.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccae7fc640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-10-30T12:18:03.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fab72ffd640 1 --2- 172.21.15.148:0/212674265 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6006c810 0x7fab6006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fab821fd640 1 --2- 172.21.15.148:0/212674265 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6006c810 0x7fab6006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fab821fd640 10 monclient: get_auth_request con 0x7fab6006c810 auth_method 0 2023-10-30T12:18:03.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fccbd564640 1 --2- 172.21.15.148:0/1248287301 >> v2:172.21.15.150:6804/4190376848 conn(0x7fcc9c0725d0 0x7fcc9c0749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fab72ffd640 10 client.4327.objecter ms_handle_connect 0x7fab6006c810 2023-10-30T12:18:03.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.437+0000 7fab821fd640 1 --2- 172.21.15.148:0/212674265 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6006c810 0x7fab6006ecd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fab78006fd0 tx=0x7fab78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.438+0000 7fccbd564640 10 monclient: get_auth_request con 0x7fcc9c0725d0 auth_method 0 2023-10-30T12:18:03.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 1 -- 172.21.15.148:0/212674265 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fab6c07bc40 con 0x7fab74034da0 2023-10-30T12:18:03.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 10 client.4327.objecter ms_dispatch 0x7fab7c000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:03.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 3 client.4327.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:03.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 3 client.4327.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:03.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 10 client.4327.objecter checking command 1 2023-10-30T12:18:03.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 1 --2- 172.21.15.148:0/212674265 >> v2:172.21.15.150:6808/2836839739 conn(0x7fab60072620 0x7fab60074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 20 client.4327.objecter _get_session s=0x7fab6c042410 osd=4 3 2023-10-30T12:18:03.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 20 client.4327.objecter put_session s=0x7fab6c042410 osd=4 3 2023-10-30T12:18:03.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 15 client.4327.objecter _session_command_op_remove -1 1 2023-10-30T12:18:03.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 20 client.4327.objecter _get_session s=0x7fab6c042410 osd=4 3 2023-10-30T12:18:03.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 20 client.4327.objecter get_session s=0x7fab6c042410 osd=4 3 2023-10-30T12:18:03.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 15 client.4327.objecter _session_command_op_assign 4 1 2023-10-30T12:18:03.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 20 client.4327.objecter put_session s=0x7fab6c042410 osd=4 4 2023-10-30T12:18:03.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 10 client.4327.objecter _send_command 1 2023-10-30T12:18:03.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 1 -- 172.21.15.148:0/212674265 --> v2:172.21.15.150:6808/2836839739 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab6c00a450 con 0x7fab60072620 2023-10-30T12:18:03.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab72ffd640 20 client.4327.objecter dump_active .. 0 homeless 2023-10-30T12:18:03.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.439+0000 7fab829fe640 1 --2- 172.21.15.148:0/212674265 >> v2:172.21.15.150:6808/2836839739 conn(0x7fab60072620 0x7fab60074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.440+0000 7fab829fe640 10 monclient: get_auth_request con 0x7fab60072620 auth_method 0 2023-10-30T12:18:03.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.440+0000 7f330e7fc640 1 -- 172.21.15.148:0/1915954374 <== osd.0 v2:172.21.15.148:6805/2395694124 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f33080052d0 con 0x7f32fc072620 2023-10-30T12:18:03.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.440+0000 7f330e7fc640 10 client.4293.objecter ms_dispatch 0x7f3318000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:18:03.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.440+0000 7f330e7fc640 10 client.4293.objecter _finish_command 1 = system:0 2023-10-30T12:18:03.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.440+0000 7f330e7fc640 20 client.4293.objecter put_session s=0x7f3308012630 osd=0 4 2023-10-30T12:18:03.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.440+0000 7f330e7fc640 15 client.4293.objecter _session_command_op_remove 0 1 2023-10-30T12:18:03.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.442+0000 7fccbd564640 1 --2- 172.21.15.148:0/1248287301 >> v2:172.21.15.150:6804/4190376848 conn(0x7fcc9c0725d0 0x7fcc9c0749f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.442+0000 7fccae7fc640 10 client.4268.objecter ms_handle_connect 0x7fcc9c0725d0 2023-10-30T12:18:03.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.444+0000 7fab72ffd640 10 client.4327.objecter ms_handle_connect 0x7fab60072620 2023-10-30T12:18:03.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.444+0000 7fab829fe640 1 --2- 172.21.15.148:0/212674265 >> v2:172.21.15.150:6808/2836839739 conn(0x7fab60072620 0x7fab60074a40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.444+0000 7fab72ffd640 1 -- 172.21.15.148:0/212674265 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab74033030 con 0x7fab74034da0 2023-10-30T12:18:03.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.444+0000 7fab72ffd640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-10-30T12:18:03.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.449+0000 7f747ffff640 1 -- 172.21.15.148:0/673158294 <== osd.3 v2:172.21.15.148:6806/2233349848 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f7490003d80 con 0x7f7484072620 2023-10-30T12:18:03.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.449+0000 7f747ffff640 10 client.4256.objecter ms_dispatch 0x7f7498000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:18:03.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.449+0000 7f747ffff640 10 client.4256.objecter _finish_command 1 = system:0 2023-10-30T12:18:03.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.449+0000 7f747ffff640 20 client.4256.objecter put_session s=0x7f74900052b0 osd=3 4 2023-10-30T12:18:03.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.449+0000 7f747ffff640 15 client.4256.objecter _session_command_op_remove 3 1 2023-10-30T12:18:03.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f7eb57fa640 1 -- 172.21.15.148:0/3037561332 <== osd.2 v2:172.21.15.148:6801/2368806934 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f7eac032950 con 0x7f7ea4072600 2023-10-30T12:18:03.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f7eb57fa640 10 client.4315.objecter ms_dispatch 0x7f7eb8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:18:03.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f7eb57fa640 10 client.4315.objecter _finish_command 1 = system:0 2023-10-30T12:18:03.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f7eb57fa640 20 client.4315.objecter put_session s=0x7f7eac032d20 osd=2 4 2023-10-30T12:18:03.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f7eb57fa640 15 client.4315.objecter _session_command_op_remove 2 1 2023-10-30T12:18:03.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 10 client.4293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:18:03.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 15 client.4293.objecter _session_command_op_assign -1 2 2023-10-30T12:18:03.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 20 client.4293.objecter _get_session s=0x7f3308012630 osd=0 3 2023-10-30T12:18:03.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 20 client.4293.objecter put_session s=0x7f3308012630 osd=0 3 2023-10-30T12:18:03.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 20 client.4293.objecter _get_session s=0x7f3308012630 osd=0 3 2023-10-30T12:18:03.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 15 client.4293.objecter _session_command_op_remove -1 2 2023-10-30T12:18:03.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 20 client.4293.objecter get_session s=0x7f3308012630 osd=0 3 2023-10-30T12:18:03.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 15 client.4293.objecter _session_command_op_assign 0 2 2023-10-30T12:18:03.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 20 client.4293.objecter put_session s=0x7f3308012630 osd=0 4 2023-10-30T12:18:03.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 10 client.4293.objecter _send_command 2 2023-10-30T12:18:03.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.455+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 --> v2:172.21.15.148:6805/2395694124 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3318126070 con 0x7f32fc072620 2023-10-30T12:18:03.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f330e7fc640 1 -- 172.21.15.148:0/1915954374 <== osd.0 v2:172.21.15.148:6805/2395694124 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3318126070 con 0x7f32fc072620 2023-10-30T12:18:03.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f330e7fc640 10 client.4293.objecter ms_dispatch 0x7f3318000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:18:03.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f330e7fc640 10 client.4293.objecter _finish_command 2 = system:0 2023-10-30T12:18:03.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f330e7fc640 20 client.4293.objecter put_session s=0x7f3308012630 osd=0 4 2023-10-30T12:18:03.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f330e7fc640 15 client.4293.objecter _session_command_op_remove 0 2 2023-10-30T12:18:03.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 10 librados: watch_flush enter 2023-10-30T12:18:03.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 10 librados: watch_flush exit 2023-10-30T12:18:03.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 10 client.4293.objecter close_session for osd.0 2023-10-30T12:18:03.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 >> v2:172.21.15.148:6805/2395694124 conn(0x7f32fc072620 msgr2=0x7f32fc074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 1 --2- 172.21.15.148:0/1915954374 >> v2:172.21.15.148:6805/2395694124 conn(0x7f32fc072620 0x7f32fc074a40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 20 client.4293.objecter put_session s=0x7f3308012630 osd=0 1 2023-10-30T12:18:03.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 20 client.4293.objecter shutdown clearing up homeless session... 2023-10-30T12:18:03.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 10 client.4293.objecter successfully canceled tick 2023-10-30T12:18:03.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 >> v2:172.21.15.148:6800/3205090179 conn(0x7f32fc06c810 msgr2=0x7f32fc06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 1 --2- 172.21.15.148:0/1915954374 >> v2:172.21.15.148:6800/3205090179 conn(0x7f32fc06c810 0x7f32fc06ecd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3314036000 tx=0x7f331403b040 comp rx=0 tx=0).stop 2023-10-30T12:18:03.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 10 monclient: shutdown 2023-10-30T12:18:03.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f331806d610 msgr2=0x7f33180ab7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 1 --2- 172.21.15.148:0/1915954374 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f331806d610 0x7f33180ab7b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f330800c480 tx=0x7f330800c940 comp rx=0 tx=0).stop 2023-10-30T12:18:03.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331e510640 1 -- 172.21.15.148:0/1915954374 reap_dead start 2023-10-30T12:18:03.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 shutdown_connections 2023-10-30T12:18:03.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 >> 172.21.15.148:0/1915954374 conn(0x7f3318069dc0 msgr2=0x7f331806a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 shutdown_connections 2023-10-30T12:18:03.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.461+0000 7f331ed11640 1 -- 172.21.15.148:0/1915954374 wait complete. 2023-10-30T12:18:03.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.462+0000 7f331ed11640 1 librados: shutdown 2023-10-30T12:18:03.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 10 client.4256.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:18:03.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 15 client.4256.objecter _session_command_op_assign -1 2 2023-10-30T12:18:03.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 20 client.4256.objecter _get_session s=0x7f74900052b0 osd=3 3 2023-10-30T12:18:03.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 20 client.4256.objecter put_session s=0x7f74900052b0 osd=3 3 2023-10-30T12:18:03.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 20 client.4256.objecter _get_session s=0x7f74900052b0 osd=3 3 2023-10-30T12:18:03.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 15 client.4256.objecter _session_command_op_remove -1 2 2023-10-30T12:18:03.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 20 client.4256.objecter get_session s=0x7f74900052b0 osd=3 3 2023-10-30T12:18:03.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 15 client.4256.objecter _session_command_op_assign 3 2 2023-10-30T12:18:03.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 20 client.4256.objecter put_session s=0x7f74900052b0 osd=3 4 2023-10-30T12:18:03.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 10 client.4256.objecter _send_command 2 2023-10-30T12:18:03.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.464+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 --> v2:172.21.15.148:6806/2233349848 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f74980af660 con 0x7f7484072620 2023-10-30T12:18:03.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.472+0000 7f9b90ff9640 1 -- 172.21.15.148:0/4136505740 <== osd.6 v2:172.21.15.150:6800/2604225934 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f9b88032880 con 0x7f9b80072f10 2023-10-30T12:18:03.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.472+0000 7f9b90ff9640 10 client.4299.objecter ms_dispatch 0x7f9b94000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:18:03.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.472+0000 7f9b90ff9640 10 client.4299.objecter _finish_command 1 = system:0 2023-10-30T12:18:03.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.472+0000 7f9b90ff9640 20 client.4299.objecter put_session s=0x7f9b88032c50 osd=6 4 2023-10-30T12:18:03.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.472+0000 7f9b90ff9640 15 client.4299.objecter _session_command_op_remove 6 1 2023-10-30T12:18:03.784 INFO:teuthology.orchestra.run.smithi148.stdout:64424509445 2023-10-30T12:18:03.784 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T12:18:03.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 10 client.4315.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:18:03.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 15 client.4315.objecter _session_command_op_assign -1 2 2023-10-30T12:18:03.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 20 client.4315.objecter _get_session s=0x7f7eac032d20 osd=2 3 2023-10-30T12:18:03.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 20 client.4315.objecter put_session s=0x7f7eac032d20 osd=2 3 2023-10-30T12:18:03.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 20 client.4315.objecter _get_session s=0x7f7eac032d20 osd=2 3 2023-10-30T12:18:03.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 15 client.4315.objecter _session_command_op_remove -1 2 2023-10-30T12:18:03.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 20 client.4315.objecter get_session s=0x7f7eac032d20 osd=2 3 2023-10-30T12:18:03.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 15 client.4315.objecter _session_command_op_assign 2 2 2023-10-30T12:18:03.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 20 client.4315.objecter put_session s=0x7f7eac032d20 osd=2 4 2023-10-30T12:18:03.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 10 client.4315.objecter _send_command 2 2023-10-30T12:18:03.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.473+0000 7f7e9effd640 1 -- 172.21.15.148:0/3037561332 --> v2:172.21.15.148:6801/2368806934 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7eb80acf30 con 0x7f7ea4072600 2023-10-30T12:18:03.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.474+0000 7f747ffff640 1 -- 172.21.15.148:0/673158294 <== osd.3 v2:172.21.15.148:6806/2233349848 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7488009200 con 0x7f7484072620 2023-10-30T12:18:03.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.474+0000 7f747ffff640 10 client.4256.objecter ms_dispatch 0x7f7498000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:18:03.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.474+0000 7f747ffff640 10 client.4256.objecter _finish_command 2 = system:0 2023-10-30T12:18:03.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.474+0000 7f747ffff640 20 client.4256.objecter put_session s=0x7f74900052b0 osd=3 4 2023-10-30T12:18:03.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.474+0000 7f747ffff640 15 client.4256.objecter _session_command_op_remove 3 2 2023-10-30T12:18:03.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 10 librados: watch_flush enter 2023-10-30T12:18:03.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 10 librados: watch_flush exit 2023-10-30T12:18:03.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 10 client.4256.objecter close_session for osd.3 2023-10-30T12:18:03.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 >> v2:172.21.15.148:6806/2233349848 conn(0x7f7484072620 msgr2=0x7f7484074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 --2- 172.21.15.148:0/673158294 >> v2:172.21.15.148:6806/2233349848 conn(0x7f7484072620 0x7f7484074a40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 20 client.4256.objecter put_session s=0x7f74900052b0 osd=3 1 2023-10-30T12:18:03.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 20 client.4256.objecter shutdown clearing up homeless session... 2023-10-30T12:18:03.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 10 client.4256.objecter successfully canceled tick 2023-10-30T12:18:03.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 >> v2:172.21.15.148:6800/3205090179 conn(0x7f748406c810 msgr2=0x7f748406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 --2- 172.21.15.148:0/673158294 >> v2:172.21.15.148:6800/3205090179 conn(0x7f748406c810 0x7f748406ecd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f748c036000 tx=0x7f748c030000 comp rx=0 tx=0).stop 2023-10-30T12:18:03.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 10 monclient: shutdown 2023-10-30T12:18:03.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f749806a030 msgr2=0x7f74980703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 --2- 172.21.15.148:0/673158294 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f749806a030 0x7f74980703b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f749000c7d0 tx=0x7f749000cc90 comp rx=0 tx=0).stop 2023-10-30T12:18:03.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f7497fff640 1 -- 172.21.15.148:0/673158294 reap_dead start 2023-10-30T12:18:03.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 shutdown_connections 2023-10-30T12:18:03.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 >> 172.21.15.148:0/673158294 conn(0x7f74980b41e0 msgr2=0x7f749813e160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 shutdown_connections 2023-10-30T12:18:03.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 -- 172.21.15.148:0/673158294 wait complete. 2023-10-30T12:18:03.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.475+0000 7f749c8e5640 1 librados: shutdown 2023-10-30T12:18:03.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.478+0000 7f59d2ffd640 1 -- 172.21.15.148:0/3896420655 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f59cc003b50 con 0x7f59dc141940 2023-10-30T12:18:03.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.479+0000 7f59d2ffd640 1 --2- 172.21.15.148:0/3896420655 >> v2:172.21.15.148:6800/3205090179 conn(0x7f59c006c810 0x7f59c006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.479+0000 7f59e223e640 1 --2- 172.21.15.148:0/3896420655 >> v2:172.21.15.148:6800/3205090179 conn(0x7f59c006c810 0x7f59c006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.479+0000 7f59e223e640 10 monclient: get_auth_request con 0x7f59c006c810 auth_method 0 2023-10-30T12:18:03.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.479+0000 7f59e223e640 1 --2- 172.21.15.148:0/3896420655 >> v2:172.21.15.148:6800/3205090179 conn(0x7f59c006c810 0x7f59c006ecd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f59d8001560 tx=0x7f59d8007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.479+0000 7f59d2ffd640 10 client.4324.objecter ms_handle_connect 0x7f59c006c810 2023-10-30T12:18:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.479+0000 7f7eb57fa640 1 -- 172.21.15.148:0/3037561332 <== osd.2 v2:172.21.15.148:6801/2368806934 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7ea8001240 con 0x7f7ea4072600 2023-10-30T12:18:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.479+0000 7f7eb57fa640 10 client.4315.objecter ms_dispatch 0x7f7eb8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:18:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.479+0000 7f7eb57fa640 10 client.4315.objecter _finish_command 2 = system:0 2023-10-30T12:18:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.479+0000 7f7eb57fa640 20 client.4315.objecter put_session s=0x7f7eac032d20 osd=2 4 2023-10-30T12:18:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.479+0000 7f7eb57fa640 15 client.4315.objecter _session_command_op_remove 2 2 2023-10-30T12:18:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 10 librados: watch_flush enter 2023-10-30T12:18:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 10 librados: watch_flush exit 2023-10-30T12:18:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 10 client.4315.objecter close_session for osd.2 2023-10-30T12:18:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 -- 172.21.15.148:0/3037561332 >> v2:172.21.15.148:6801/2368806934 conn(0x7f7ea4072600 msgr2=0x7f7ea4074a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 --2- 172.21.15.148:0/3037561332 >> v2:172.21.15.148:6801/2368806934 conn(0x7f7ea4072600 0x7f7ea4074a20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 20 client.4315.objecter put_session s=0x7f7eac032d20 osd=2 1 2023-10-30T12:18:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 20 client.4315.objecter shutdown clearing up homeless session... 2023-10-30T12:18:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 10 client.4315.objecter successfully canceled tick 2023-10-30T12:18:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 -- 172.21.15.148:0/3037561332 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7ea406c7f0 msgr2=0x7f7ea406ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 --2- 172.21.15.148:0/3037561332 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7ea406c7f0 0x7f7ea406ecb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0002070 tx=0x7f7eb0001660 comp rx=0 tx=0).stop 2023-10-30T12:18:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 10 monclient: shutdown 2023-10-30T12:18:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 -- 172.21.15.148:0/3037561332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806dd80 msgr2=0x7f7eb8135f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 --2- 172.21.15.148:0/3037561332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806dd80 0x7f7eb8135f60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0097d0 tx=0x7f7eac0046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7ebd546640 1 -- 172.21.15.148:0/3037561332 reap_dead start 2023-10-30T12:18:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 -- 172.21.15.148:0/3037561332 shutdown_connections 2023-10-30T12:18:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 -- 172.21.15.148:0/3037561332 >> 172.21.15.148:0/3037561332 conn(0x7f7eb8069dc0 msgr2=0x7f7eb8140f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 -- 172.21.15.148:0/3037561332 shutdown_connections 2023-10-30T12:18:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 -- 172.21.15.148:0/3037561332 wait complete. 2023-10-30T12:18:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.480+0000 7f7e9effd640 1 librados: shutdown 2023-10-30T12:18:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 1 -- 172.21.15.148:0/3896420655 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f59cc0581d0 con 0x7f59dc141940 2023-10-30T12:18:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 10 client.4324.objecter ms_dispatch 0x7f59dc000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 3 client.4324.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 3 client.4324.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 10 client.4324.objecter checking command 1 2023-10-30T12:18:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 1 --2- 172.21.15.148:0/3896420655 >> v2:172.21.15.148:6813/912412250 conn(0x7f59c0072620 0x7f59c0074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 20 client.4324.objecter _get_session s=0x7f59cc0039e0 osd=1 3 2023-10-30T12:18:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 20 client.4324.objecter put_session s=0x7f59cc0039e0 osd=1 3 2023-10-30T12:18:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 15 client.4324.objecter _session_command_op_remove -1 1 2023-10-30T12:18:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 20 client.4324.objecter _get_session s=0x7f59cc0039e0 osd=1 3 2023-10-30T12:18:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 20 client.4324.objecter get_session s=0x7f59cc0039e0 osd=1 3 2023-10-30T12:18:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 15 client.4324.objecter _session_command_op_assign 1 1 2023-10-30T12:18:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 20 client.4324.objecter put_session s=0x7f59cc0039e0 osd=1 4 2023-10-30T12:18:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 10 client.4324.objecter _send_command 1 2023-10-30T12:18:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 1 -- 172.21.15.148:0/3896420655 --> v2:172.21.15.148:6813/912412250 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59cc0035f0 con 0x7f59c0072620 2023-10-30T12:18:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.484+0000 7f59d2ffd640 20 client.4324.objecter dump_active .. 0 homeless 2023-10-30T12:18:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.485+0000 7f59e2a3f640 1 --2- 172.21.15.148:0/3896420655 >> v2:172.21.15.148:6813/912412250 conn(0x7f59c0072620 0x7f59c0074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.485+0000 7f59e2a3f640 10 monclient: get_auth_request con 0x7f59c0072620 auth_method 0 2023-10-30T12:18:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.485+0000 7f59d2ffd640 1 -- 172.21.15.148:0/3896420655 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59cc053e10 con 0x7f59dc141940 2023-10-30T12:18:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.485+0000 7f59d2ffd640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-10-30T12:18:03.810 INFO:teuthology.orchestra.run.smithi148.stdout:64424509445 2023-10-30T12:18:03.810 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T12:18:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.488+0000 7f59e2a3f640 1 --2- 172.21.15.148:0/3896420655 >> v2:172.21.15.148:6813/912412250 conn(0x7f59c0072620 0x7f59c0074a40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.488+0000 7f59d2ffd640 10 client.4324.objecter ms_handle_connect 0x7f59c0072620 2023-10-30T12:18:03.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 10 client.4299.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:18:03.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 15 client.4299.objecter _session_command_op_assign -1 2 2023-10-30T12:18:03.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 20 client.4299.objecter _get_session s=0x7f9b88032c50 osd=6 3 2023-10-30T12:18:03.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 20 client.4299.objecter put_session s=0x7f9b88032c50 osd=6 3 2023-10-30T12:18:03.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 20 client.4299.objecter _get_session s=0x7f9b88032c50 osd=6 3 2023-10-30T12:18:03.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 15 client.4299.objecter _session_command_op_remove -1 2 2023-10-30T12:18:03.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 20 client.4299.objecter get_session s=0x7f9b88032c50 osd=6 3 2023-10-30T12:18:03.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 15 client.4299.objecter _session_command_op_assign 6 2 2023-10-30T12:18:03.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 20 client.4299.objecter put_session s=0x7f9b88032c50 osd=6 4 2023-10-30T12:18:03.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 10 client.4299.objecter _send_command 2 2023-10-30T12:18:03.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.495+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 --> v2:172.21.15.150:6800/2604225934 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9b940ad620 con 0x7f9b80072f10 2023-10-30T12:18:03.814 INFO:teuthology.orchestra.run.smithi148.stdout:64424509446 2023-10-30T12:18:03.815 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:18:03.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.501+0000 7fccae7fc640 1 -- 172.21.15.148:0/1248287301 <== osd.5 v2:172.21.15.150:6804/4190376848 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fccb804e6d0 con 0x7fcc9c0725d0 2023-10-30T12:18:03.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.501+0000 7fccae7fc640 10 client.4268.objecter ms_dispatch 0x7fccb8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:18:03.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.501+0000 7fccae7fc640 10 client.4268.objecter _finish_command 1 = system:0 2023-10-30T12:18:03.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.501+0000 7fccae7fc640 20 client.4268.objecter put_session s=0x7fccb80721c0 osd=5 4 2023-10-30T12:18:03.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.501+0000 7fccae7fc640 15 client.4268.objecter _session_command_op_remove 5 1 2023-10-30T12:18:03.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b90ff9640 1 -- 172.21.15.148:0/4136505740 <== osd.6 v2:172.21.15.150:6800/2604225934 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9b8c001c30 con 0x7f9b80072f10 2023-10-30T12:18:03.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b90ff9640 10 client.4299.objecter ms_dispatch 0x7f9b94000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:18:03.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b90ff9640 10 client.4299.objecter _finish_command 2 = system:0 2023-10-30T12:18:03.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b90ff9640 20 client.4299.objecter put_session s=0x7f9b88032c50 osd=6 4 2023-10-30T12:18:03.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b90ff9640 15 client.4299.objecter _session_command_op_remove 6 2 2023-10-30T12:18:03.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 10 librados: watch_flush enter 2023-10-30T12:18:03.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 10 librados: watch_flush exit 2023-10-30T12:18:03.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 10 client.4299.objecter close_session for osd.6 2023-10-30T12:18:03.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 >> v2:172.21.15.150:6800/2604225934 conn(0x7f9b80072f10 msgr2=0x7f9b800753b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 1 --2- 172.21.15.148:0/4136505740 >> v2:172.21.15.150:6800/2604225934 conn(0x7f9b80072f10 0x7f9b800753b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 20 client.4299.objecter put_session s=0x7f9b88032c50 osd=6 1 2023-10-30T12:18:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 20 client.4299.objecter shutdown clearing up homeless session... 2023-10-30T12:18:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 10 client.4299.objecter successfully canceled tick 2023-10-30T12:18:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 >> v2:172.21.15.148:6800/3205090179 conn(0x7f9b8006c7c0 msgr2=0x7f9b8006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 1 --2- 172.21.15.148:0/4136505740 >> v2:172.21.15.148:6800/3205090179 conn(0x7f9b8006c7c0 0x7f9b8006ec80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009a70 tx=0x7f9b84009110 comp rx=0 tx=0).stop 2023-10-30T12:18:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 10 monclient: shutdown 2023-10-30T12:18:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b94142370 msgr2=0x7f9b941d7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.506+0000 7f9b9a973640 1 --2- 172.21.15.148:0/4136505740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b94142370 0x7f9b941d7440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b9406bcb0 tx=0x7f9b880046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.507+0000 7f9b98f10640 1 -- 172.21.15.148:0/4136505740 reap_dead start 2023-10-30T12:18:03.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.507+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 shutdown_connections 2023-10-30T12:18:03.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.507+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 >> 172.21.15.148:0/4136505740 conn(0x7f9b94069dc0 msgr2=0x7f9b9413d410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.507+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 shutdown_connections 2023-10-30T12:18:03.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.507+0000 7f9b9a973640 1 -- 172.21.15.148:0/4136505740 wait complete. 2023-10-30T12:18:03.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.507+0000 7f9b9a973640 1 librados: shutdown 2023-10-30T12:18:03.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.509+0000 7fab72ffd640 1 -- 172.21.15.148:0/212674265 <== osd.4 v2:172.21.15.150:6808/2836839739 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fab6c00a450 con 0x7fab60072620 2023-10-30T12:18:03.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.509+0000 7fab72ffd640 10 client.4327.objecter ms_dispatch 0x7fab7c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:18:03.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.509+0000 7fab72ffd640 10 client.4327.objecter _finish_command 1 = system:0 2023-10-30T12:18:03.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.509+0000 7fab72ffd640 20 client.4327.objecter put_session s=0x7fab6c042410 osd=4 4 2023-10-30T12:18:03.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.509+0000 7fab72ffd640 15 client.4327.objecter _session_command_op_remove 4 1 2023-10-30T12:18:03.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 10 client.4268.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:18:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 15 client.4268.objecter _session_command_op_assign -1 2 2023-10-30T12:18:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 20 client.4268.objecter _get_session s=0x7fccb80721c0 osd=5 3 2023-10-30T12:18:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 20 client.4268.objecter put_session s=0x7fccb80721c0 osd=5 3 2023-10-30T12:18:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 20 client.4268.objecter _get_session s=0x7fccb80721c0 osd=5 3 2023-10-30T12:18:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 15 client.4268.objecter _session_command_op_remove -1 2 2023-10-30T12:18:03.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 20 client.4268.objecter get_session s=0x7fccb80721c0 osd=5 3 2023-10-30T12:18:03.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 15 client.4268.objecter _session_command_op_assign 5 2 2023-10-30T12:18:03.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 20 client.4268.objecter put_session s=0x7fccb80721c0 osd=5 4 2023-10-30T12:18:03.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 10 client.4268.objecter _send_command 2 2023-10-30T12:18:03.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.516+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 --> v2:172.21.15.150:6804/4190376848 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fccb00374e0 con 0x7fcc9c0725d0 2023-10-30T12:18:03.827 INFO:teuthology.orchestra.run.smithi148.stdout:55834574854 2023-10-30T12:18:03.828 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T12:18:03.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.522+0000 7fccae7fc640 1 -- 172.21.15.148:0/1248287301 <== osd.5 v2:172.21.15.150:6804/4190376848 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcca800a940 con 0x7fcc9c0725d0 2023-10-30T12:18:03.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.522+0000 7fccae7fc640 10 client.4268.objecter ms_dispatch 0x7fccb8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:18:03.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.522+0000 7fccae7fc640 10 client.4268.objecter _finish_command 2 = system:0 2023-10-30T12:18:03.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.522+0000 7fccae7fc640 20 client.4268.objecter put_session s=0x7fccb80721c0 osd=5 4 2023-10-30T12:18:03.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.522+0000 7fccae7fc640 15 client.4268.objecter _session_command_op_remove 5 2 2023-10-30T12:18:03.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 10 librados: watch_flush enter 2023-10-30T12:18:03.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 10 librados: watch_flush exit 2023-10-30T12:18:03.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 10 client.4268.objecter close_session for osd.5 2023-10-30T12:18:03.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 >> v2:172.21.15.150:6804/4190376848 conn(0x7fcc9c0725d0 msgr2=0x7fcc9c0749f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 --2- 172.21.15.148:0/1248287301 >> v2:172.21.15.150:6804/4190376848 conn(0x7fcc9c0725d0 0x7fcc9c0749f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 20 client.4268.objecter put_session s=0x7fccb80721c0 osd=5 1 2023-10-30T12:18:03.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 20 client.4268.objecter shutdown clearing up homeless session... 2023-10-30T12:18:03.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 10 client.4268.objecter successfully canceled tick 2023-10-30T12:18:03.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcc9c06c7c0 msgr2=0x7fcc9c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 --2- 172.21.15.148:0/1248287301 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcc9c06c7c0 0x7fcc9c06ec80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fccb4036000 tx=0x7fccb4030000 comp rx=0 tx=0).stop 2023-10-30T12:18:03.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 10 monclient: shutdown 2023-10-30T12:18:03.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb00c7db0 msgr2=0x7fccb015cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 --2- 172.21.15.148:0/1248287301 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fccb00c7db0 0x7fccb015cdf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fccb80726d0 tx=0x7fccb8067a20 comp rx=0 tx=0).stop 2023-10-30T12:18:03.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbe566640 1 -- 172.21.15.148:0/1248287301 reap_dead start 2023-10-30T12:18:03.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 shutdown_connections 2023-10-30T12:18:03.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 >> 172.21.15.148:0/1248287301 conn(0x7fccb0007180 msgr2=0x7fccb0038d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 shutdown_connections 2023-10-30T12:18:03.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 -- 172.21.15.148:0/1248287301 wait complete. 2023-10-30T12:18:03.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.523+0000 7fccbed67640 1 librados: shutdown 2023-10-30T12:18:03.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 10 client.4327.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:18:03.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 15 client.4327.objecter _session_command_op_assign -1 2 2023-10-30T12:18:03.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 20 client.4327.objecter _get_session s=0x7fab6c042410 osd=4 3 2023-10-30T12:18:03.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 20 client.4327.objecter put_session s=0x7fab6c042410 osd=4 3 2023-10-30T12:18:03.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 20 client.4327.objecter _get_session s=0x7fab6c042410 osd=4 3 2023-10-30T12:18:03.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 15 client.4327.objecter _session_command_op_remove -1 2 2023-10-30T12:18:03.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 20 client.4327.objecter get_session s=0x7fab6c042410 osd=4 3 2023-10-30T12:18:03.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 15 client.4327.objecter _session_command_op_assign 4 2 2023-10-30T12:18:03.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 20 client.4327.objecter put_session s=0x7fab6c042410 osd=4 4 2023-10-30T12:18:03.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 10 client.4327.objecter _send_command 2 2023-10-30T12:18:03.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.524+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 --> v2:172.21.15.150:6808/2836839739 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fab74035180 con 0x7fab60072620 2023-10-30T12:18:03.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab72ffd640 1 -- 172.21.15.148:0/212674265 <== osd.4 v2:172.21.15.150:6808/2836839739 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fab7c04f050 con 0x7fab60072620 2023-10-30T12:18:03.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab72ffd640 10 client.4327.objecter ms_dispatch 0x7fab7c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:18:03.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab72ffd640 10 client.4327.objecter _finish_command 2 = system:0 2023-10-30T12:18:03.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab72ffd640 20 client.4327.objecter put_session s=0x7fab6c042410 osd=4 4 2023-10-30T12:18:03.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab72ffd640 15 client.4327.objecter _session_command_op_remove 4 2 2023-10-30T12:18:03.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab84461640 10 librados: watch_flush enter 2023-10-30T12:18:03.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab84461640 10 librados: watch_flush exit 2023-10-30T12:18:03.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab84461640 10 client.4327.objecter close_session for osd.4 2023-10-30T12:18:03.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 >> v2:172.21.15.150:6808/2836839739 conn(0x7fab60072620 msgr2=0x7fab60074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab84461640 1 --2- 172.21.15.148:0/212674265 >> v2:172.21.15.150:6808/2836839739 conn(0x7fab60072620 0x7fab60074a40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab84461640 20 client.4327.objecter put_session s=0x7fab6c042410 osd=4 1 2023-10-30T12:18:03.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab84461640 20 client.4327.objecter shutdown clearing up homeless session... 2023-10-30T12:18:03.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.530+0000 7fab84461640 10 client.4327.objecter successfully canceled tick 2023-10-30T12:18:03.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6006c810 msgr2=0x7fab6006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 1 --2- 172.21.15.148:0/212674265 >> v2:172.21.15.148:6800/3205090179 conn(0x7fab6006c810 0x7fab6006ecd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fab78006fd0 tx=0x7fab78008040 comp rx=0 tx=0).stop 2023-10-30T12:18:03.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 10 monclient: shutdown 2023-10-30T12:18:03.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab74034da0 msgr2=0x7fab7402ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 1 --2- 172.21.15.148:0/212674265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab74034da0 0x7fab7402ed70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab6c03a040 tx=0x7fab6c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab829fe640 1 -- 172.21.15.148:0/212674265 reap_dead start 2023-10-30T12:18:03.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 shutdown_connections 2023-10-30T12:18:03.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 >> 172.21.15.148:0/212674265 conn(0x7fab740061f0 msgr2=0x7fab740c7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 shutdown_connections 2023-10-30T12:18:03.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 1 -- 172.21.15.148:0/212674265 wait complete. 2023-10-30T12:18:03.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.531+0000 7fab84461640 1 librados: shutdown 2023-10-30T12:18:03.847 INFO:teuthology.orchestra.run.smithi148.stdout:64424509446 2023-10-30T12:18:03.847 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:18:03.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.537+0000 7f59d2ffd640 1 -- 172.21.15.148:0/3896420655 <== osd.1 v2:172.21.15.148:6813/912412250 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f59cc0035f0 con 0x7f59c0072620 2023-10-30T12:18:03.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.537+0000 7f59d2ffd640 10 client.4324.objecter ms_dispatch 0x7f59dc000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:18:03.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.537+0000 7f59d2ffd640 10 client.4324.objecter _finish_command 1 = system:0 2023-10-30T12:18:03.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.537+0000 7f59d2ffd640 20 client.4324.objecter put_session s=0x7f59cc0039e0 osd=1 4 2023-10-30T12:18:03.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.537+0000 7f59d2ffd640 15 client.4324.objecter _session_command_op_remove 1 1 2023-10-30T12:18:03.849 INFO:teuthology.orchestra.run.smithi148.stdout:64424509446 2023-10-30T12:18:03.850 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:18:03.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 10 client.4324.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:18:03.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 15 client.4324.objecter _session_command_op_assign -1 2 2023-10-30T12:18:03.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 20 client.4324.objecter _get_session s=0x7f59cc0039e0 osd=1 3 2023-10-30T12:18:03.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 20 client.4324.objecter put_session s=0x7f59cc0039e0 osd=1 3 2023-10-30T12:18:03.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 20 client.4324.objecter _get_session s=0x7f59cc0039e0 osd=1 3 2023-10-30T12:18:03.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 15 client.4324.objecter _session_command_op_remove -1 2 2023-10-30T12:18:03.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 20 client.4324.objecter get_session s=0x7f59cc0039e0 osd=1 3 2023-10-30T12:18:03.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 15 client.4324.objecter _session_command_op_assign 1 2 2023-10-30T12:18:03.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 20 client.4324.objecter put_session s=0x7f59cc0039e0 osd=1 4 2023-10-30T12:18:03.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 10 client.4324.objecter _send_command 2 2023-10-30T12:18:03.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.551+0000 7f59d0ff9640 1 -- 172.21.15.148:0/3896420655 --> v2:172.21.15.148:6813/912412250 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f59c40011c0 con 0x7f59c0072620 2023-10-30T12:18:03.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d2ffd640 1 -- 172.21.15.148:0/3896420655 <== osd.1 v2:172.21.15.148:6813/912412250 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f59c40011c0 con 0x7f59c0072620 2023-10-30T12:18:03.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d2ffd640 10 client.4324.objecter ms_dispatch 0x7f59dc000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:18:03.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d2ffd640 10 client.4324.objecter _finish_command 2 = system:0 2023-10-30T12:18:03.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d2ffd640 20 client.4324.objecter put_session s=0x7f59cc0039e0 osd=1 4 2023-10-30T12:18:03.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d2ffd640 15 client.4324.objecter _session_command_op_remove 1 2 2023-10-30T12:18:03.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 10 librados: watch_flush enter 2023-10-30T12:18:03.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 10 librados: watch_flush exit 2023-10-30T12:18:03.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 10 client.4324.objecter close_session for osd.1 2023-10-30T12:18:03.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 1 -- 172.21.15.148:0/3896420655 >> v2:172.21.15.148:6813/912412250 conn(0x7f59c0072620 msgr2=0x7f59c0074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 1 --2- 172.21.15.148:0/3896420655 >> v2:172.21.15.148:6813/912412250 conn(0x7f59c0072620 0x7f59c0074a40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 20 client.4324.objecter put_session s=0x7f59cc0039e0 osd=1 1 2023-10-30T12:18:03.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 20 client.4324.objecter shutdown clearing up homeless session... 2023-10-30T12:18:03.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 10 client.4324.objecter successfully canceled tick 2023-10-30T12:18:03.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 1 -- 172.21.15.148:0/3896420655 >> v2:172.21.15.148:6800/3205090179 conn(0x7f59c006c810 msgr2=0x7f59c006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 1 --2- 172.21.15.148:0/3896420655 >> v2:172.21.15.148:6800/3205090179 conn(0x7f59c006c810 0x7f59c006ecd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f59d8001560 tx=0x7f59d8007990 comp rx=0 tx=0).stop 2023-10-30T12:18:03.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 10 monclient: shutdown 2023-10-30T12:18:03.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 1 -- 172.21.15.148:0/3896420655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc141940 msgr2=0x7f59dc1333d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 1 --2- 172.21.15.148:0/3896420655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc141940 0x7f59dc1333d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00c410 tx=0x7f59cc00c8d0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59e2a3f640 1 -- 172.21.15.148:0/3896420655 reap_dead start 2023-10-30T12:18:03.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 1 -- 172.21.15.148:0/3896420655 shutdown_connections 2023-10-30T12:18:03.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 1 -- 172.21.15.148:0/3896420655 >> 172.21.15.148:0/3896420655 conn(0x7f59dc06c660 msgr2=0x7f59dc0a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 1 -- 172.21.15.148:0/3896420655 shutdown_connections 2023-10-30T12:18:03.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.557+0000 7f59d0ff9640 1 -- 172.21.15.148:0/3896420655 wait complete. 2023-10-30T12:18:03.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.558+0000 7f59d0ff9640 1 librados: shutdown 2023-10-30T12:18:03.860 INFO:teuthology.orchestra.run.smithi148.stdout:64424509446 2023-10-30T12:18:03.861 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T12:18:03.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.868+0000 7f88e2ffd640 1 -- 172.21.15.148:0/4211863896 <== osd.7 v2:172.21.15.150:6812/3463178370 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f88ec03d600 con 0x7f88d4072620 2023-10-30T12:18:03.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.868+0000 7f88e2ffd640 10 client.4318.objecter ms_dispatch 0x7f88f4000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:18:03.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.868+0000 7f88e2ffd640 10 client.4318.objecter _finish_command 1 = system:0 2023-10-30T12:18:03.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.868+0000 7f88e2ffd640 20 client.4318.objecter put_session s=0x7f88ec03db10 osd=7 4 2023-10-30T12:18:03.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.868+0000 7f88e2ffd640 15 client.4318.objecter _session_command_op_remove 7 1 2023-10-30T12:18:03.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:18:03.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 15 client.4318.objecter _session_command_op_assign -1 2 2023-10-30T12:18:03.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 20 client.4318.objecter _get_session s=0x7f88ec03db10 osd=7 3 2023-10-30T12:18:03.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 20 client.4318.objecter put_session s=0x7f88ec03db10 osd=7 3 2023-10-30T12:18:03.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 20 client.4318.objecter _get_session s=0x7f88ec03db10 osd=7 3 2023-10-30T12:18:03.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 15 client.4318.objecter _session_command_op_remove -1 2 2023-10-30T12:18:03.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 20 client.4318.objecter get_session s=0x7f88ec03db10 osd=7 3 2023-10-30T12:18:03.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 15 client.4318.objecter _session_command_op_assign 7 2 2023-10-30T12:18:03.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 20 client.4318.objecter put_session s=0x7f88ec03db10 osd=7 4 2023-10-30T12:18:03.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 10 client.4318.objecter _send_command 2 2023-10-30T12:18:03.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.895+0000 7f88f359e640 1 -- 172.21.15.148:0/4211863896 --> v2:172.21.15.150:6812/3463178370 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f88f406cf60 con 0x7f88d4072620 2023-10-30T12:18:03.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.901+0000 7f88e2ffd640 1 -- 172.21.15.148:0/4211863896 <== osd.7 v2:172.21.15.150:6812/3463178370 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f88e4001580 con 0x7f88d4072620 2023-10-30T12:18:03.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.901+0000 7f88e2ffd640 10 client.4318.objecter ms_dispatch 0x7f88f4000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:18:03.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.901+0000 7f88e2ffd640 10 client.4318.objecter _finish_command 2 = system:0 2023-10-30T12:18:03.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.901+0000 7f88e2ffd640 20 client.4318.objecter put_session s=0x7f88ec03db10 osd=7 4 2023-10-30T12:18:03.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.901+0000 7f88e2ffd640 15 client.4318.objecter _session_command_op_remove 7 2 2023-10-30T12:18:03.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.903+0000 7f88e0ff9640 10 librados: watch_flush enter 2023-10-30T12:18:03.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.903+0000 7f88e0ff9640 10 librados: watch_flush exit 2023-10-30T12:18:03.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.904+0000 7f88e0ff9640 10 client.4318.objecter close_session for osd.7 2023-10-30T12:18:03.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.904+0000 7f88e0ff9640 1 -- 172.21.15.148:0/4211863896 >> v2:172.21.15.150:6812/3463178370 conn(0x7f88d4072620 msgr2=0x7f88d4074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.904+0000 7f88e0ff9640 1 --2- 172.21.15.148:0/4211863896 >> v2:172.21.15.150:6812/3463178370 conn(0x7f88d4072620 0x7f88d4074a40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.904+0000 7f88e0ff9640 20 client.4318.objecter put_session s=0x7f88ec03db10 osd=7 1 2023-10-30T12:18:03.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.904+0000 7f88e0ff9640 20 client.4318.objecter shutdown clearing up homeless session... 2023-10-30T12:18:03.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.904+0000 7f88e0ff9640 10 client.4318.objecter successfully canceled tick 2023-10-30T12:18:03.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.905+0000 7f88e0ff9640 1 -- 172.21.15.148:0/4211863896 >> v2:172.21.15.148:6800/3205090179 conn(0x7f88d406c810 msgr2=0x7f88d406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.905+0000 7f88e0ff9640 1 --2- 172.21.15.148:0/4211863896 >> v2:172.21.15.148:6800/3205090179 conn(0x7f88d406c810 0x7f88d406ecd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009a40 tx=0x7f88e80091c0 comp rx=0 tx=0).stop 2023-10-30T12:18:03.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.905+0000 7f88e0ff9640 10 monclient: shutdown 2023-10-30T12:18:03.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.906+0000 7f88e0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:03.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.906+0000 7f88e0ff9640 1 -- 172.21.15.148:0/4211863896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88f4140d10 msgr2=0x7f88f4137e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:03.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.906+0000 7f88e0ff9640 1 --2- 172.21.15.148:0/4211863896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88f4140d10 0x7f88f4137e90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f88ec033000 tx=0x7f88ec004a90 comp rx=0 tx=0).stop 2023-10-30T12:18:03.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.907+0000 7f88f2d9d640 1 -- 172.21.15.148:0/4211863896 reap_dead start 2023-10-30T12:18:03.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.911+0000 7f88e0ff9640 1 -- 172.21.15.148:0/4211863896 shutdown_connections 2023-10-30T12:18:03.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.911+0000 7f88e0ff9640 1 -- 172.21.15.148:0/4211863896 >> 172.21.15.148:0/4211863896 conn(0x7f88f4069dc0 msgr2=0x7f88f4140300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:03.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.912+0000 7f88e0ff9640 1 -- 172.21.15.148:0/4211863896 shutdown_connections 2023-10-30T12:18:03.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.912+0000 7f88e0ff9640 1 -- 172.21.15.148:0/4211863896 wait complete. 2023-10-30T12:18:03.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:03.913+0000 7f88e0ff9640 1 librados: shutdown 2023-10-30T12:18:03.939 INFO:teuthology.orchestra.run.smithi148.stdout:64424509446 2023-10-30T12:18:03.940 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T12:18:04.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.043+0000 7f8517b7c640 10 monclient: get_monmap_and_config 2023-10-30T12:18:04.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.043+0000 7f8517b7c640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.043+0000 7f8517b7c640 10 monclient: monmap: 2023-10-30T12:18:04.045 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.046 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.046 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:04.044181+0000 2023-10-30T12:18:04.046 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:04.044181+0000 2023-10-30T12:18:04.046 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.047 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.047 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:04.047 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:04.047 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:04.047 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:04.048 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:04.048 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.048+0000 7f8517b7c640 1 Processor -- start 2023-10-30T12:18:04.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.048+0000 7f8517b7c640 1 -- start start 2023-10-30T12:18:04.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.048+0000 7f8517b7c640 10 monclient: init 2023-10-30T12:18:04.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.048+0000 7f57a1f33640 10 monclient: get_monmap_and_config 2023-10-30T12:18:04.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.048+0000 7f57a1f33640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.048+0000 7f57a1f33640 10 monclient: monmap: 2023-10-30T12:18:04.051 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.051 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.051 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:04.049388+0000 2023-10-30T12:18:04.051 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:04.049388+0000 2023-10-30T12:18:04.051 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.052 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.052 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:04.052 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:04.052 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:04.053 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:04.053 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:04.053 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.049+0000 7f8517b7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.049+0000 7f8517b7c640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-10-30T12:18:04.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.049+0000 7f8517b7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85100a9750 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:04.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.049+0000 7f8517b7c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f85100adb20 0x7f85100adf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.049+0000 7f8517b7c640 10 monclient(hunting): picked mon.noname-a con 0x7f85100adb20 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:04.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.049+0000 7f8517b7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85100ae490 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:04.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.049+0000 7f8517b7c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851013e910 con 0x7f85100ae490 2023-10-30T12:18:04.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.049+0000 7f8517b7c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85100ad620 con 0x7f85100a9750 2023-10-30T12:18:04.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.049+0000 7f8517b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.049+0000 7f8517b7c640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85100670d0 con 0x7f85100adb20 2023-10-30T12:18:04.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.050+0000 7f8517b7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.050+0000 7f8517b7c640 10 monclient(hunting): authenticate will time out at 1576.254909s 2023-10-30T12:18:04.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.051+0000 7f8515918640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f85100a9750 0x7f851013e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:48252/0 (socket says 172.21.15.148:48252) 2023-10-30T12:18:04.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.051+0000 7f8515918640 1 -- 172.21.15.148:0/2287172660 learned_addr learned my addr 172.21.15.148:0/2287172660 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:18:04.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.051+0000 7f802759e640 10 monclient: get_monmap_and_config 2023-10-30T12:18:04.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.051+0000 7f802759e640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.051+0000 7f802759e640 10 monclient: monmap: 2023-10-30T12:18:04.059 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.059 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.059 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:04.052461+0000 2023-10-30T12:18:04.059 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:04.052461+0000 2023-10-30T12:18:04.060 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.060 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.060 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:04.060 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:04.061 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:04.061 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:04.061 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:04.061 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.052+0000 7f057b634640 10 monclient: get_monmap_and_config 2023-10-30T12:18:04.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.052+0000 7f057b634640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.052+0000 7f057b634640 10 monclient: monmap: 2023-10-30T12:18:04.062 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.063 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.063 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:04.053595+0000 2023-10-30T12:18:04.063 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:04.053595+0000 2023-10-30T12:18:04.063 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.063 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.064 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:04.064 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:04.064 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:04.064 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:04.064 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:04.065 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.052+0000 7f8515117640 1 --2- 172.21.15.148:0/2287172660 >> v2:172.21.15.148:3300/0 conn(0x7f85100adb20 0x7f85100adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.052+0000 7f8515117640 10 monclient(hunting): get_auth_request con 0x7f85100adb20 auth_method 0 2023-10-30T12:18:04.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.052+0000 7f8515117640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.052+0000 7f8515117640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.052+0000 7f8515117640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.053+0000 7f802759e640 1 Processor -- start 2023-10-30T12:18:04.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.053+0000 7f802759e640 1 -- start start 2023-10-30T12:18:04.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.053+0000 7f802759e640 10 monclient: init 2023-10-30T12:18:04.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.053+0000 7f57a1f33640 1 Processor -- start 2023-10-30T12:18:04.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.053+0000 7f57a1f33640 1 -- start start 2023-10-30T12:18:04.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.053+0000 7f57a1f33640 10 monclient: init 2023-10-30T12:18:04.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-10-30T12:18:04.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f802806bc50 0x7f8028070550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 10 monclient(hunting): picked mon.noname-c con 0x7f802806bc50 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:04.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f802806ac90 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:04.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8028140d10 0x7f8028141160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 10 monclient(hunting): picked mon.noname-b con 0x7f8028140d10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80281416a0 con 0x7f802806ac90 2023-10-30T12:18:04.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80280670d0 con 0x7f802806bc50 2023-10-30T12:18:04.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802813e760 con 0x7f8028140d10 2023-10-30T12:18:04.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f802759e640 10 monclient(hunting): authenticate will time out at 1576.259180s 2023-10-30T12:18:04.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f57a1f33640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f57a1f33640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-10-30T12:18:04.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.053+0000 7f8c68850640 10 monclient: get_monmap_and_config 2023-10-30T12:18:04.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.053+0000 7f8c68850640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.053+0000 7f8c68850640 10 monclient: monmap: 2023-10-30T12:18:04.074 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.075 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.075 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:04.054841+0000 2023-10-30T12:18:04.075 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:04.054841+0000 2023-10-30T12:18:04.075 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.076 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.076 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:04.076 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:04.076 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:04.076 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:04.077 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:04.077 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f57a1f33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57940354f0 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:04.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.054+0000 7f57a1f33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57940c1b20 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:04.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f57a1f33640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f579402b730 0x7f57940c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f57a1f33640 10 monclient(hunting): picked mon.noname-c con 0x7f579402b730 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:04.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f802659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f802806bc50 0x7f8028070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f57a1f33640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579402be10 con 0x7f57940354f0 2023-10-30T12:18:04.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f802659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f802806bc50 0x7f8028070550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48746/0 (socket says 172.21.15.148:48746) 2023-10-30T12:18:04.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f8514916640 1 -- 172.21.15.148:0/2287172660 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974258626 0 0) 0x7f8500001b20 con 0x7f85100a9750 2023-10-30T12:18:04.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f8514916640 10 monclient(hunting): _negotiate 2023-10-30T12:18:04.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f8514916640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f8514916640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f8514916640 10 monclient(hunting): my global_id is 4330 2023-10-30T12:18:04.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f8514916640 1 -- 172.21.15.148:0/2287172660 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f8003520 con 0x7f85100a9750 2023-10-30T12:18:04.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f802659c640 1 -- 172.21.15.148:0/384706442 learned_addr learned my addr 172.21.15.148:0/384706442 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f802659c640 10 monclient(hunting): get_auth_request con 0x7f802806bc50 auth_method 0 2023-10-30T12:18:04.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f57a1f33640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579402cd40 con 0x7f57940c1b20 2023-10-30T12:18:04.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f57a1f33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f57a1f33640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5794008f00 con 0x7f579402b730 2023-10-30T12:18:04.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f57a1f33640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f57a1f33640 10 monclient(hunting): authenticate will time out at 1576.260606s 2023-10-30T12:18:04.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f802659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f802659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f802659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f802659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f802659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f802659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8026d9d640 1 --2- 172.21.15.148:0/384706442 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8028140d10 0x7f8028141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8026d9d640 10 monclient(hunting): get_auth_request con 0x7f8028140d10 auth_method 0 2023-10-30T12:18:04.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8026d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8026d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8026d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f057b634640 1 Processor -- start 2023-10-30T12:18:04.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f057b634640 1 -- start start 2023-10-30T12:18:04.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f057b634640 10 monclient: init 2023-10-30T12:18:04.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f261e4ca640 10 monclient: get_monmap_and_config 2023-10-30T12:18:04.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f261e4ca640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f261e4ca640 10 monclient: monmap: 2023-10-30T12:18:04.090 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.090 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.090 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:04.056793+0000 2023-10-30T12:18:04.091 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:04.056793+0000 2023-10-30T12:18:04.091 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.091 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.091 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:04.091 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:04.092 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:04.092 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:04.092 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:04.092 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8515117640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8515117640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8515117640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f261e4ca640 1 Processor -- start 2023-10-30T12:18:04.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f579b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f57940354f0 0x7f57940c1410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53022/0 (socket says 172.21.15.148:53022) 2023-10-30T12:18:04.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f579b7fe640 1 -- 172.21.15.148:0/2840205417 learned_addr learned my addr 172.21.15.148:0/2840205417 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:18:04.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-10-30T12:18:04.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f057406bb80 0x7f057406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 10 monclient(hunting): picked mon.noname-c con 0x7f057406bb80 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:04.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f05740721b0 0x7f05740afd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f261e4ca640 1 -- start start 2023-10-30T12:18:04.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f261e4ca640 10 monclient: init 2023-10-30T12:18:04.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f8c68850640 1 Processor -- start 2023-10-30T12:18:04.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f8c68850640 1 -- start start 2023-10-30T12:18:04.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.055+0000 7f8c68850640 10 monclient: init 2023-10-30T12:18:04.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-10-30T12:18:04.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c5c034d70 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:04.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8c5c02ac00 0x7f8c5c0c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 10 monclient(hunting): picked mon.noname-c con 0x7f8c5c02ac00 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:04.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8c5c0cc7b0 0x7f8c5c0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 10 monclient(hunting): picked mon.noname-a con 0x7f8c5c0cc7b0 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:04.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5c02b1b0 con 0x7f8c5c034d70 2023-10-30T12:18:04.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c5c008780 con 0x7f8c5c0cc7b0 2023-10-30T12:18:04.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c5c035150 con 0x7f8c5c02ac00 2023-10-30T12:18:04.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.056+0000 7f8c68850640 10 monclient(hunting): authenticate will time out at 1576.261752s 2023-10-30T12:18:04.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 10 monclient(hunting): picked mon.noname-b con 0x7f05740721b0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f05741451d0 0x7f0574145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 10 monclient(hunting): picked mon.noname-a con 0x7f05741451d0 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:04.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05740670a0 con 0x7f05741451d0 2023-10-30T12:18:04.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0574071a90 con 0x7f057406bb80 2023-10-30T12:18:04.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057406bf60 con 0x7f05740721b0 2023-10-30T12:18:04.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.057+0000 7f057b634640 10 monclient(hunting): authenticate will time out at 1576.262834s 2023-10-30T12:18:04.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f8026d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f8026d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f8026d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f802659c640 10 monclient(hunting): handle_auth_done global_id 4333 payload 274 2023-10-30T12:18:04.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f802659c640 1 -- 172.21.15.148:0/384706442 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8028140d10 msgr2=0x7f8028141160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:04.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f802659c640 1 --2- 172.21.15.148:0/384706442 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8028140d10 0x7f8028141160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f802659c640 1 -- 172.21.15.148:0/384706442 >> v1:172.21.15.148:6789/0 conn(0x7f802806ac90 legacy=0x7f802813e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:04.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f802659c640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f802659c640 1 monclient: found mon.noname-c 2023-10-30T12:18:04.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f802659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f802659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f802659c640 1 -- 172.21.15.148:0/384706442 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8028071940 con 0x7f802806bc50 2023-10-30T12:18:04.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-10-30T12:18:04.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2610034d70 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:04.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f26100c3c80 0x7f26100c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 10 monclient(hunting): picked mon.noname-a con 0x7f26100c3c80 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:04.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 0x7f26100ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 10 monclient(hunting): picked mon.noname-b con 0x7f26100cc580 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261002cd40 con 0x7f2610034d70 2023-10-30T12:18:04.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2610008780 con 0x7f26100c3c80 2023-10-30T12:18:04.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261002cb40 con 0x7f26100cc580 2023-10-30T12:18:04.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f261e4ca640 10 monclient(hunting): authenticate will time out at 1576.263258s 2023-10-30T12:18:04.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.058+0000 7f0578bcf640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f05740721b0 0x7f05740afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.059+0000 7f0578bcf640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f05740721b0 0x7f05740afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54472/0 (socket says 172.21.15.148:54472) 2023-10-30T12:18:04.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.059+0000 7f0578bcf640 1 -- 172.21.15.148:0/2521721392 learned_addr learned my addr 172.21.15.148:0/2521721392 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.059+0000 7f0578bcf640 10 monclient(hunting): get_auth_request con 0x7f05740721b0 auth_method 0 2023-10-30T12:18:04.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.059+0000 7f0578bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.059+0000 7f0578bcf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.059+0000 7f0578bcf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.060+0000 7f8515117640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2023-10-30T12:18:04.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.060+0000 7f8515117640 1 -- 172.21.15.148:0/2287172660 >> v1:172.21.15.150:6789/0 conn(0x7f85100a9750 legacy=0x7f851013e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.060+0000 7f8515117640 1 -- 172.21.15.148:0/2287172660 >> v1:172.21.15.148:6789/0 conn(0x7f85100ae490 legacy=0x7f85101480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.061+0000 7f8515117640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.061+0000 7f8515117640 1 monclient: found mon.noname-a 2023-10-30T12:18:04.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.061+0000 7f8515117640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.061+0000 7f8515117640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.061+0000 7f8515117640 1 -- 172.21.15.148:0/2287172660 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85100aabc0 con 0x7f85100adb20 2023-10-30T12:18:04.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.062+0000 7f0578bcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.062+0000 7f0578bcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.062+0000 7f0578bcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.062+0000 7f802659c640 1 --2- 172.21.15.148:0/384706442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802806bc50 0x7f8028070550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8018009870 tx=0x7f801802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=622161ecf2c400c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f0578bcf640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2023-10-30T12:18:04.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f0578bcf640 1 -- 172.21.15.148:0/2521721392 >> v2:172.21.15.150:3300/0 conn(0x7f057406bb80 msgr2=0x7f057406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:04.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f0578bcf640 1 --2- 172.21.15.148:0/2521721392 >> v2:172.21.15.150:3300/0 conn(0x7f057406bb80 0x7f057406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f0578bcf640 1 -- 172.21.15.148:0/2521721392 >> v2:172.21.15.148:3300/0 conn(0x7f05741451d0 msgr2=0x7f0574145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:04.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f0578bcf640 1 --2- 172.21.15.148:0/2521721392 >> v2:172.21.15.148:3300/0 conn(0x7f05741451d0 0x7f0574145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f0578bcf640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f0578bcf640 1 monclient: found mon.noname-b 2023-10-30T12:18:04.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f0578bcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f0578bcf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f0578bcf640 1 -- 172.21.15.148:0/2521721392 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057406fd60 con 0x7f05740721b0 2023-10-30T12:18:04.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.066+0000 7f579bfff640 1 --2- 172.21.15.148:0/2840205417 >> v2:172.21.15.150:3300/0 conn(0x7f579402b730 0x7f57940c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.067+0000 7f579bfff640 10 monclient(hunting): get_auth_request con 0x7f579402b730 auth_method 0 2023-10-30T12:18:04.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.067+0000 7f579bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.067+0000 7f579bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.067+0000 7f579bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.067+0000 7f8c6259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8c5c02ac00 0x7f8c5c0c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.067+0000 7f8515117640 1 --2- 172.21.15.148:0/2287172660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85100adb20 0x7f85100adf50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f85040097a0 tx=0x7f850402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3bca7321e4c039 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.067+0000 7f579bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.068+0000 7f579bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.068+0000 7f579bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.068+0000 7f8c6259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8c5c02ac00 0x7f8c5c0c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48762/0 (socket says 172.21.15.148:48762) 2023-10-30T12:18:04.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.068+0000 7f8c6259c640 1 -- 172.21.15.148:0/1295377272 learned_addr learned my addr 172.21.15.148:0/1295377272 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.068+0000 7f8c6259c640 10 monclient(hunting): get_auth_request con 0x7f8c5c02ac00 auth_method 0 2023-10-30T12:18:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.068+0000 7f8c6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.068+0000 7f8c6259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.068+0000 7f8c6259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.069+0000 7f261dcc9640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 0x7f26100ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.069+0000 7f8c6359e640 1 --2- 172.21.15.148:0/1295377272 >> v2:172.21.15.148:3300/0 conn(0x7f8c5c0cc7b0 0x7f8c5c0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.070+0000 7f261dcc9640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 0x7f26100ce970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54482/0 (socket says 172.21.15.148:54482) 2023-10-30T12:18:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.070+0000 7f261dcc9640 1 -- 172.21.15.148:0/3335495682 learned_addr learned my addr 172.21.15.148:0/3335495682 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.070+0000 7f261dcc9640 10 monclient(hunting): get_auth_request con 0x7f26100cc580 auth_method 0 2023-10-30T12:18:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.070+0000 7f261dcc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.070+0000 7f261dcc9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.070+0000 7f261dcc9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.070+0000 7f0578bcf640 1 --2- 172.21.15.148:0/2521721392 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f05740721b0 0x7f05740afd50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f057406b660 tx=0x7f056c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e50f42dd05ad21b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.070+0000 7f8c6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.070+0000 7f8c6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.070+0000 7f8c6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 1 -- 172.21.15.148:0/384706442 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018031640 con 0x7f802806bc50 2023-10-30T12:18:04.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:04.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 10 monclient: dump: 2023-10-30T12:18:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.130 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.130 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.130 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.130 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.131 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f261ccc7640 1 --2- 172.21.15.148:0/3335495682 >> v2:172.21.15.148:3300/0 conn(0x7f26100c3c80 0x7f26100c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802759e640 5 monclient: authenticate success, global_id 4333 2023-10-30T12:18:04.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:04.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f261dcc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f261dcc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f261dcc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 1 -- 172.21.15.148:0/384706442 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80180317b0 con 0x7f802806bc50 2023-10-30T12:18:04.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802759e640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:04.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802759e640 4 set_mon_vals no callback set 2023-10-30T12:18:04.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 1 -- 172.21.15.148:0/384706442 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018031b50 con 0x7f802806bc50 2023-10-30T12:18:04.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:04.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f802559a640 10 monclient: dump: 2023-10-30T12:18:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.137 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.137 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.137 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.137 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f261ccc7640 10 monclient(hunting): get_auth_request con 0x7f26100c3c80 auth_method 0 2023-10-30T12:18:04.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f261ccc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f261ccc7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.071+0000 7f261ccc7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f8c6359e640 10 monclient(hunting): get_auth_request con 0x7f8c5c0cc7b0 auth_method 0 2023-10-30T12:18:04.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f8c6359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f8c6359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f8c6359e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f802759e640 10 monclient: shutdown 2023-10-30T12:18:04.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f802759e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f802759e640 1 -- 172.21.15.148:0/384706442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802806bc50 msgr2=0x7f8028070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f802759e640 1 --2- 172.21.15.148:0/384706442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802806bc50 0x7f8028070550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8018009870 tx=0x7f801802b8f0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f802759e640 1 -- 172.21.15.148:0/384706442 shutdown_connections 2023-10-30T12:18:04.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f802759e640 1 --2- 172.21.15.148:0/384706442 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8028140d10 0x7f8028141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f802759e640 1 --2- 172.21.15.148:0/384706442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802806bc50 0x7f8028070550 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f802759e640 1 -- 172.21.15.148:0/384706442 >> 172.21.15.148:0/384706442 conn(0x7f8028069dc0 msgr2=0x7f802806a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f802759e640 1 -- 172.21.15.148:0/384706442 shutdown_connections 2023-10-30T12:18:04.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f802759e640 1 -- 172.21.15.148:0/384706442 wait complete. 2023-10-30T12:18:04.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f579bfff640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2023-10-30T12:18:04.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f579bfff640 1 -- 172.21.15.148:0/2840205417 >> v1:172.21.15.150:6789/0 conn(0x7f57940c1b20 legacy=0x7f57940c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:04.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f579bfff640 1 -- 172.21.15.148:0/2840205417 >> v1:172.21.15.148:6789/0 conn(0x7f57940354f0 legacy=0x7f57940c1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f579bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f579bfff640 1 monclient: found mon.noname-c 2023-10-30T12:18:04.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f579bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f579bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f579bfff640 1 -- 172.21.15.148:0/2840205417 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57940c8460 con 0x7f579402b730 2023-10-30T12:18:04.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f8c6359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f8c6359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.072+0000 7f8c6359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.073+0000 7f802759e640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.073+0000 7f802759e640 10 monclient: monmap: 2023-10-30T12:18:04.147 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.147 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:04.147 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:04.147 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:04.148 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.148 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:04.148 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:04.148 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:04.149 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:04.149 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.073+0000 7f802759e640 1 librados: starting msgr at 2023-10-30T12:18:04.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.073+0000 7f802759e640 1 librados: starting objecter 2023-10-30T12:18:04.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 1 Processor -- start 2023-10-30T12:18:04.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 1 -- start start 2023-10-30T12:18:04.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 1 librados: setting wanted keys 2023-10-30T12:18:04.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 1 librados: calling monclient init 2023-10-30T12:18:04.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient: init 2023-10-30T12:18:04.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:04.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f802806ac90 0x7f802813f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient(hunting): picked mon.noname-c con 0x7f802806ac90 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f802806bc50 0x7f802812b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient(hunting): picked mon.noname-a con 0x7f802806bc50 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:04.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8028140d10 0x7f802812b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient(hunting): picked mon.noname-b con 0x7f8028140d10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:04.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8028070550 con 0x7f802806bc50 2023-10-30T12:18:04.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802812cc40 con 0x7f8028140d10 2023-10-30T12:18:04.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80280a90f0 con 0x7f802806ac90 2023-10-30T12:18:04.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.074+0000 7f802759e640 10 monclient(hunting): authenticate will time out at 1576.279550s 2023-10-30T12:18:04.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.075+0000 7f8c6259c640 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2023-10-30T12:18:04.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.075+0000 7f8c6259c640 1 -- 172.21.15.148:0/1295377272 >> v2:172.21.15.148:3300/0 conn(0x7f8c5c0cc7b0 msgr2=0x7f8c5c0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:04.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.075+0000 7f8c6259c640 1 --2- 172.21.15.148:0/1295377272 >> v2:172.21.15.148:3300/0 conn(0x7f8c5c0cc7b0 0x7f8c5c0c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.075+0000 7f8c6259c640 1 -- 172.21.15.148:0/1295377272 >> v1:172.21.15.150:6789/0 conn(0x7f8c5c034d70 legacy=0x7f8c5c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.075+0000 7f8c6259c640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.075+0000 7f8c6259c640 1 monclient: found mon.noname-c 2023-10-30T12:18:04.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.075+0000 7f8c6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.075+0000 7f8c6259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.075+0000 7f8c6259c640 1 -- 172.21.15.148:0/1295377272 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c02c2c0 con 0x7f8c5c02ac00 2023-10-30T12:18:04.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.076+0000 7f261dcc9640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2023-10-30T12:18:04.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.076+0000 7f261dcc9640 1 -- 172.21.15.148:0/3335495682 >> v2:172.21.15.148:3300/0 conn(0x7f26100c3c80 msgr2=0x7f26100c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:04.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.076+0000 7f261dcc9640 1 --2- 172.21.15.148:0/3335495682 >> v2:172.21.15.148:3300/0 conn(0x7f26100c3c80 0x7f26100c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.076+0000 7f261dcc9640 1 -- 172.21.15.148:0/3335495682 >> v1:172.21.15.148:6789/0 conn(0x7f2610034d70 legacy=0x7f26100326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.076+0000 7f261dcc9640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.076+0000 7f261dcc9640 1 monclient: found mon.noname-b 2023-10-30T12:18:04.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.076+0000 7f261dcc9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.076+0000 7f261dcc9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.076+0000 7f261dcc9640 1 -- 172.21.15.148:0/3335495682 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26100c17c0 con 0x7f26100cc580 2023-10-30T12:18:04.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.077+0000 7f579bfff640 1 --2- 172.21.15.148:0/2840205417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f579402b730 0x7f57940c7f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5790006fd0 tx=0x7f579002ccf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6647cee7cfa3f693 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f261dcc9640 1 --2- 172.21.15.148:0/3335495682 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 0x7f26100ce970 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2608007360 tx=0x7f260802db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8f37057e8e08ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.080+0000 7f056bfff640 1 -- 172.21.15.148:0/2521721392 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c02c9e0 con 0x7f05740721b0 2023-10-30T12:18:04.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.080+0000 7f056bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:04.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 10 monclient: dump: 2023-10-30T12:18:04.160 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.160 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.160 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.160 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.162 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 1 -- 172.21.15.148:0/2521721392 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c02cb50 con 0x7f05740721b0 2023-10-30T12:18:04.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 1 -- 172.21.15.148:0/2521721392 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c0343f0 con 0x7f05740721b0 2023-10-30T12:18:04.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:04.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f056bfff640 10 monclient: dump: 2023-10-30T12:18:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.165 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f057b634640 5 monclient: authenticate success, global_id 4274 2023-10-30T12:18:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f057b634640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:04.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f057b634640 4 set_mon_vals no callback set 2023-10-30T12:18:04.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f579a7fc640 1 -- 172.21.15.148:0/2840205417 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579002daa0 con 0x7f579402b730 2023-10-30T12:18:04.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f579a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f579a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:04.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f579a7fc640 10 monclient: dump: 2023-10-30T12:18:04.167 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.167 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.167 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.167 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.169 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f579a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f579a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f579a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f57a1f33640 5 monclient: authenticate success, global_id 4336 2023-10-30T12:18:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.081+0000 7f57a1f33640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:04.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f8c6259c640 1 --2- 172.21.15.148:0/1295377272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c02ac00 0x7f8c5c0c8e90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c02bf10 tx=0x7f8c5802b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=740aff1cabbb1303 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 10 monclient: shutdown 2023-10-30T12:18:04.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 1 -- 172.21.15.148:0/2521721392 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f05740721b0 msgr2=0x7f05740afd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 1 --2- 172.21.15.148:0/2521721392 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f05740721b0 0x7f05740afd50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f057406b660 tx=0x7f056c02be40 comp rx=0 tx=0).stop 2023-10-30T12:18:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 1 -- 172.21.15.148:0/2521721392 shutdown_connections 2023-10-30T12:18:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 1 --2- 172.21.15.148:0/2521721392 >> v2:172.21.15.148:3300/0 conn(0x7f05741451d0 0x7f0574145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 1 --2- 172.21.15.148:0/2521721392 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f05740721b0 0x7f05740afd50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 1 --2- 172.21.15.148:0/2521721392 >> v2:172.21.15.150:3300/0 conn(0x7f057406bb80 0x7f057406cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 1 -- 172.21.15.148:0/2521721392 >> 172.21.15.148:0/2521721392 conn(0x7f05740b54e0 msgr2=0x7f05740b58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 1 -- 172.21.15.148:0/2521721392 shutdown_connections 2023-10-30T12:18:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.082+0000 7f057b634640 1 -- 172.21.15.148:0/2521721392 wait complete. 2023-10-30T12:18:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.083+0000 7f057b634640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.083+0000 7f057b634640 10 monclient: monmap: 2023-10-30T12:18:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:04.175 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:04.175 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:04.175 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:04.175 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.084+0000 7f8dd4826640 10 monclient: get_monmap_and_config 2023-10-30T12:18:04.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.084+0000 7f8dd4826640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.084+0000 7f8dd4826640 10 monclient: monmap: 2023-10-30T12:18:04.176 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.176 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:04.085460+0000 2023-10-30T12:18:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:04.085460+0000 2023-10-30T12:18:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:04.178 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:04.178 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:04.178 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:04.178 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.084+0000 7f057b634640 1 librados: starting msgr at 2023-10-30T12:18:04.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.084+0000 7f057b634640 1 librados: starting objecter 2023-10-30T12:18:04.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.084+0000 7f057b634640 1 Processor -- start 2023-10-30T12:18:04.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.084+0000 7f057b634640 1 -- start start 2023-10-30T12:18:04.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.084+0000 7f057b634640 1 librados: setting wanted keys 2023-10-30T12:18:04.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.084+0000 7f057b634640 1 librados: calling monclient init 2023-10-30T12:18:04.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.084+0000 7f057b634640 10 monclient: init 2023-10-30T12:18:04.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.085+0000 7f8dd4826640 1 Processor -- start 2023-10-30T12:18:04.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.085+0000 7f8dd4826640 1 -- start start 2023-10-30T12:18:04.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.085+0000 7f8dd4826640 10 monclient: init 2023-10-30T12:18:04.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.085+0000 7f057b634640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.085+0000 7f057b634640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:04.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.085+0000 7f057b634640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f057406bb80 0x7f05740b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 10 monclient(hunting): picked mon.noname-c con 0x7f057406bb80 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f05740721b0 0x7f05740b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 10 monclient(hunting): picked mon.noname-a con 0x7f05740721b0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:04.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05741451d0 0x7f057413b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 10 monclient(hunting): picked mon.noname-b con 0x7f05741451d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:04.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05740abae0 con 0x7f05740721b0 2023-10-30T12:18:04.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05740b4f30 con 0x7f05741451d0 2023-10-30T12:18:04.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05740b31f0 con 0x7f057406bb80 2023-10-30T12:18:04.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-10-30T12:18:04.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8dd00b7c10 0x7f8dd00a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 10 monclient(hunting): picked mon.noname-c con 0x7f8dd00b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:04.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f057b634640 10 monclient(hunting): authenticate will time out at 1576.291257s 2023-10-30T12:18:04.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8dd0141940 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:04.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8dd013e730 0x7f8dd013eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 10 monclient(hunting): picked mon.noname-b con 0x7f8dd013e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd00a9350 con 0x7f8dd0141940 2023-10-30T12:18:04.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd00670a0 con 0x7f8dd00b7c10 2023-10-30T12:18:04.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00b7ff0 con 0x7f8dd013e730 2023-10-30T12:18:04.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8025d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f802806bc50 0x7f802812b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.086+0000 7f8dd4826640 10 monclient(hunting): authenticate will time out at 1576.291623s 2023-10-30T12:18:04.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f8025d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f802806bc50 0x7f802812b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59964/0 (socket says 172.21.15.148:59964) 2023-10-30T12:18:04.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f8025d9b640 1 -- 172.21.15.148:0/2700389748 learned_addr learned my addr 172.21.15.148:0/2700389748 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f8025d9b640 10 monclient(hunting): get_auth_request con 0x7f802806bc50 auth_method 0 2023-10-30T12:18:04.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f8025d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f8025d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f8025d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f579a7fc640 1 -- 172.21.15.148:0/2840205417 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5790009030 con 0x7f579402b730 2023-10-30T12:18:04.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f579a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f579a7fc640 1 -- 172.21.15.148:0/2840205417 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579002de30 con 0x7f579402b730 2023-10-30T12:18:04.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f579a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f579a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:04.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f579a7fc640 10 monclient: dump: 2023-10-30T12:18:04.195 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.195 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.195 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.196 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.196 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.196 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.196 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.196 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.197 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.197 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f57a1f33640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:04.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f57a1f33640 4 set_mon_vals no callback set 2023-10-30T12:18:04.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 1 -- 172.21.15.148:0/3335495682 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2608008df0 con 0x7f26100cc580 2023-10-30T12:18:04.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:04.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 10 monclient: dump: 2023-10-30T12:18:04.199 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.199 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.199 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.199 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.199 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.200 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.200 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.200 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.200 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.200 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 1 -- 172.21.15.148:0/3335495682 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f260800a0d0 con 0x7f26100cc580 2023-10-30T12:18:04.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 1 -- 172.21.15.148:0/3335495682 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260800a470 con 0x7f26100cc580 2023-10-30T12:18:04.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:04.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f260ffff640 10 monclient: dump: 2023-10-30T12:18:04.202 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.202 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.203 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.203 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.203 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.203 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.204 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.204 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.204 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.204 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f261e4ca640 5 monclient: authenticate success, global_id 4277 2023-10-30T12:18:04.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f261e4ca640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:04.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.087+0000 7f261e4ca640 4 set_mon_vals no callback set 2023-10-30T12:18:04.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8025d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8025d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8025d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 1 -- 172.21.15.148:0/1295377272 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c58031880 con 0x7f8c5c02ac00 2023-10-30T12:18:04.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:04.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 10 monclient: dump: 2023-10-30T12:18:04.207 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.207 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.207 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.207 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.207 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.208 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.208 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.208 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.208 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.208 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 1 -- 172.21.15.148:0/1295377272 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c580319f0 con 0x7f8c5c02ac00 2023-10-30T12:18:04.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 1 -- 172.21.15.148:0/1295377272 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c58031d90 con 0x7f8c5c02ac00 2023-10-30T12:18:04.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:04.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c61d9b640 10 monclient: dump: 2023-10-30T12:18:04.210 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.211 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.211 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.211 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.211 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.212 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.212 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.212 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.212 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.212 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f57a1f33640 10 monclient: shutdown 2023-10-30T12:18:04.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f57a1f33640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f57a1f33640 1 -- 172.21.15.148:0/2840205417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f579402b730 msgr2=0x7f57940c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f57a1f33640 1 --2- 172.21.15.148:0/2840205417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f579402b730 0x7f57940c7f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5790006fd0 tx=0x7f579002ccf0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f57a1f33640 1 -- 172.21.15.148:0/2840205417 shutdown_connections 2023-10-30T12:18:04.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f57a1f33640 1 --2- 172.21.15.148:0/2840205417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f579402b730 0x7f57940c7f20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f57a1f33640 1 -- 172.21.15.148:0/2840205417 >> 172.21.15.148:0/2840205417 conn(0x7f5794006a70 msgr2=0x7f5794006e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f261e4ca640 10 monclient: shutdown 2023-10-30T12:18:04.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c68850640 5 monclient: authenticate success, global_id 4339 2023-10-30T12:18:04.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c68850640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:04.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8c68850640 4 set_mon_vals no callback set 2023-10-30T12:18:04.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f261e4ca640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f261e4ca640 1 -- 172.21.15.148:0/3335495682 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 msgr2=0x7f26100ce970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f261e4ca640 1 --2- 172.21.15.148:0/3335495682 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 0x7f26100ce970 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2608007360 tx=0x7f260802db90 comp rx=0 tx=0).stop 2023-10-30T12:18:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f05793d0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f057406bb80 0x7f05740b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f57a1f33640 1 -- 172.21.15.148:0/2840205417 shutdown_connections 2023-10-30T12:18:04.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f57a1f33640 1 -- 172.21.15.148:0/2840205417 wait complete. 2023-10-30T12:18:04.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f261e4ca640 1 -- 172.21.15.148:0/3335495682 shutdown_connections 2023-10-30T12:18:04.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f261e4ca640 1 --2- 172.21.15.148:0/3335495682 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 0x7f26100ce970 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f261e4ca640 1 --2- 172.21.15.148:0/3335495682 >> v2:172.21.15.148:3300/0 conn(0x7f26100c3c80 0x7f26100c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f261e4ca640 1 -- 172.21.15.148:0/3335495682 >> 172.21.15.148:0/3335495682 conn(0x7f26100061c0 msgr2=0x7f26100065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8c68850640 10 monclient: shutdown 2023-10-30T12:18:04.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8c68850640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8c68850640 1 -- 172.21.15.148:0/1295377272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c02ac00 msgr2=0x7f8c5c0c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8c68850640 1 --2- 172.21.15.148:0/1295377272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c02ac00 0x7f8c5c0c8e90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c02bf10 tx=0x7f8c5802b920 comp rx=0 tx=0).stop 2023-10-30T12:18:04.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8c68850640 1 -- 172.21.15.148:0/1295377272 shutdown_connections 2023-10-30T12:18:04.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8c68850640 1 --2- 172.21.15.148:0/1295377272 >> v2:172.21.15.148:3300/0 conn(0x7f8c5c0cc7b0 0x7f8c5c0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8c68850640 1 --2- 172.21.15.148:0/1295377272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c02ac00 0x7f8c5c0c8e90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8c68850640 1 -- 172.21.15.148:0/1295377272 >> 172.21.15.148:0/1295377272 conn(0x7f8c5c0061c0 msgr2=0x7f8c5c0065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8dce59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8dd00b7c10 0x7f8dd00a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f261e4ca640 1 -- 172.21.15.148:0/3335495682 shutdown_connections 2023-10-30T12:18:04.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8026d9d640 1 --2- 172.21.15.148:0/2700389748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8028140d10 0x7f802812b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8026d9d640 10 monclient(hunting): get_auth_request con 0x7f8028140d10 auth_method 0 2023-10-30T12:18:04.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8026d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8026d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f8026d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.088+0000 7f802659c640 1 --2- 172.21.15.148:0/2700389748 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f802806ac90 0x7f802813f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f802659c640 10 monclient(hunting): get_auth_request con 0x7f802806ac90 auth_method 0 2023-10-30T12:18:04.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f802659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f802659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f802659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f57a1f33640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f8dce59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8dd00b7c10 0x7f8dd00a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48798/0 (socket says 172.21.15.148:48798) 2023-10-30T12:18:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f8dce59c640 1 -- 172.21.15.148:0/910375795 learned_addr learned my addr 172.21.15.148:0/910375795 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f8dce59c640 10 monclient(hunting): get_auth_request con 0x7f8dd00b7c10 auth_method 0 2023-10-30T12:18:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f8dce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f8dce59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f8dce59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f57a1f33640 10 monclient: monmap: 2023-10-30T12:18:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:04.228 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f261e4ca640 1 -- 172.21.15.148:0/3335495682 wait complete. 2023-10-30T12:18:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f261e4ca640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f261e4ca640 10 monclient: monmap: 2023-10-30T12:18:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:04.231 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f261e4ca640 1 librados: starting msgr at 2023-10-30T12:18:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f261e4ca640 1 librados: starting objecter 2023-10-30T12:18:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f261e4ca640 1 Processor -- start 2023-10-30T12:18:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f05793d0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f057406bb80 0x7f05740b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54512/0 (socket says 172.21.15.148:54512) 2023-10-30T12:18:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f05793d0640 1 -- 172.21.15.148:0/2860908784 learned_addr learned my addr 172.21.15.148:0/2860908784 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f05793d0640 10 monclient(hunting): get_auth_request con 0x7f057406bb80 auth_method 0 2023-10-30T12:18:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f05793d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f05793d0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f05793d0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f57a1f33640 1 librados: starting msgr at 2023-10-30T12:18:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f261e4ca640 1 -- start start 2023-10-30T12:18:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f261e4ca640 1 librados: setting wanted keys 2023-10-30T12:18:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f261e4ca640 1 librados: calling monclient init 2023-10-30T12:18:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f261e4ca640 10 monclient: init 2023-10-30T12:18:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f8dced9d640 1 --2- 172.21.15.148:0/910375795 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8dd013e730 0x7f8dd013eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f57a1f33640 1 librados: starting objecter 2023-10-30T12:18:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8c68850640 1 -- 172.21.15.148:0/1295377272 shutdown_connections 2023-10-30T12:18:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.089+0000 7f8c68850640 1 -- 172.21.15.148:0/1295377272 wait complete. 2023-10-30T12:18:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f8c68850640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.090+0000 7f8c68850640 10 monclient: monmap: 2023-10-30T12:18:04.238 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.238 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:04.238 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:04.238 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:04.240 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f8c68850640 1 librados: starting msgr at 2023-10-30T12:18:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f8c68850640 1 librados: starting objecter 2023-10-30T12:18:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f8c68850640 1 Processor -- start 2023-10-30T12:18:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f8c68850640 1 -- start start 2023-10-30T12:18:04.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f8c68850640 1 librados: setting wanted keys 2023-10-30T12:18:04.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f8c68850640 1 librados: calling monclient init 2023-10-30T12:18:04.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f8c68850640 10 monclient: init 2023-10-30T12:18:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f57a1f33640 1 Processor -- start 2023-10-30T12:18:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f57a1f33640 1 -- start start 2023-10-30T12:18:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f57a1f33640 1 librados: setting wanted keys 2023-10-30T12:18:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f261e4ca640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8dce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8dce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8dce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f261e4ca640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:18:04.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f802659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f802659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.091+0000 7f802659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8026d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8026d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8026d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f261e4ca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2610034d70 0x7f26100c0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:18:04.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8c5c02ac00 0x7f8c5c030970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 10 monclient(hunting): picked mon.noname-c con 0x7f8c5c02ac00 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c034d70 0x7f8c5c18f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 10 monclient(hunting): picked mon.noname-b con 0x7f8c5c034d70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:04.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c5c0cc7b0 0x7f8c5c18fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 10 monclient(hunting): picked mon.noname-a con 0x7f8c5c0cc7b0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:04.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c02a950 con 0x7f8c5c0cc7b0 2023-10-30T12:18:04.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c0d0710 con 0x7f8c5c034d70 2023-10-30T12:18:04.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c032580 con 0x7f8c5c02ac00 2023-10-30T12:18:04.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f8c68850640 10 monclient(hunting): authenticate will time out at 1576.297573s 2023-10-30T12:18:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f261e4ca640 10 monclient(hunting): picked mon.noname-b con 0x7f2610034d70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f57a1f33640 1 librados: calling monclient init 2023-10-30T12:18:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f57a1f33640 10 monclient: init 2023-10-30T12:18:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.092+0000 7f261e4ca640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26100c3c80 0x7f26100c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 10 monclient(hunting): picked mon.noname-a con 0x7f26100c3c80 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 0x7f26100bbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f05793d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f05793d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f05793d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 10 monclient(hunting): picked mon.noname-c con 0x7f26100cc580 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2610000df0 con 0x7f26100c3c80 2023-10-30T12:18:04.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f8025d9b640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2023-10-30T12:18:04.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f8025d9b640 1 -- 172.21.15.148:0/2700389748 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f802806ac90 msgr2=0x7f802813f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f8025d9b640 1 --2- 172.21.15.148:0/2700389748 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f802806ac90 0x7f802813f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f8025d9b640 1 -- 172.21.15.148:0/2700389748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8028140d10 msgr2=0x7f802812b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f8025d9b640 1 --2- 172.21.15.148:0/2700389748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8028140d10 0x7f802812b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f8025d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f8025d9b640 1 monclient: found mon.noname-a 2023-10-30T12:18:04.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f8025d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f8025d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f8025d9b640 1 -- 172.21.15.148:0/2700389748 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8028140170 con 0x7f802806bc50 2023-10-30T12:18:04.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:04.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f579402b730 0x7f57940b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 10 monclient(hunting): picked mon.noname-c con 0x7f579402b730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57940354f0 0x7f579415cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 10 monclient(hunting): picked mon.noname-a con 0x7f57940354f0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:04.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57940c1b20 0x7f579415d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26100c8cb0 con 0x7f2610034d70 2023-10-30T12:18:04.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 10 monclient(hunting): picked mon.noname-b con 0x7f57940c1b20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:04.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579402b300 con 0x7f57940354f0 2023-10-30T12:18:04.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940c5250 con 0x7f57940c1b20 2023-10-30T12:18:04.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57940b4d80 con 0x7f579402b730 2023-10-30T12:18:04.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f57a1f33640 10 monclient(hunting): authenticate will time out at 1576.298456s 2023-10-30T12:18:04.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26100c10e0 con 0x7f26100cc580 2023-10-30T12:18:04.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.093+0000 7f261e4ca640 10 monclient(hunting): authenticate will time out at 1576.298801s 2023-10-30T12:18:04.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.094+0000 7f8dce59c640 10 monclient(hunting): handle_auth_done global_id 4345 payload 274 2023-10-30T12:18:04.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.094+0000 7f8dce59c640 1 -- 172.21.15.148:0/910375795 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8dd013e730 msgr2=0x7f8dd013eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:04.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.094+0000 7f8dce59c640 1 --2- 172.21.15.148:0/910375795 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8dd013e730 0x7f8dd013eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.094+0000 7f8dce59c640 1 -- 172.21.15.148:0/910375795 >> v1:172.21.15.148:6789/0 conn(0x7f8dd0141940 legacy=0x7f8dd0141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:04.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.094+0000 7f8dce59c640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.094+0000 7f8dce59c640 1 monclient: found mon.noname-c 2023-10-30T12:18:04.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.094+0000 7f8dce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.094+0000 7f8dce59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.094+0000 7f8dce59c640 1 -- 172.21.15.148:0/910375795 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd00aa6a0 con 0x7f8dd00b7c10 2023-10-30T12:18:04.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.095+0000 7f8025d9b640 1 --2- 172.21.15.148:0/2700389748 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f802806bc50 0x7f802812b470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f801c00b3f0 tx=0x7f801c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.095+0000 7f8016ffd640 10 client.?.objecter ms_handle_connect 0x7f802806bc50 2023-10-30T12:18:04.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.095+0000 7f8016ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:04.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.097+0000 7f0578bcf640 1 --2- 172.21.15.148:0/2860908784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f05740721b0 0x7f05740b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.097+0000 7f0578bcf640 10 monclient(hunting): get_auth_request con 0x7f05740721b0 auth_method 0 2023-10-30T12:18:04.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.097+0000 7f0578bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.097+0000 7f0578bcf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.097+0000 7f0578bcf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.099+0000 7f0578bcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.099+0000 7f0578bcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.099+0000 7f0578bcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.100+0000 7f579affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57940354f0 0x7f579415cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.100+0000 7f8c6259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c034d70 0x7f8c5c18f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.100+0000 7f8dce59c640 1 --2- 172.21.15.148:0/910375795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd00b7c10 0x7f8dd00a8aa0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009870 tx=0x7f8dc002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9073ba64f9d7efe9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.101+0000 7f261d4c8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2610034d70 0x7f26100c0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.101+0000 7f8c6259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c034d70 0x7f8c5c18f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48806/0 (socket says 172.21.15.148:48806) 2023-10-30T12:18:04.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.101+0000 7f8c6259c640 1 -- 172.21.15.148:0/323870612 learned_addr learned my addr 172.21.15.148:0/323870612 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.101+0000 7f8c6259c640 10 monclient(hunting): get_auth_request con 0x7f8c5c034d70 auth_method 0 2023-10-30T12:18:04.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.101+0000 7f8c6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.101+0000 7f8c6259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.101+0000 7f8c6259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.102+0000 7f261d4c8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2610034d70 0x7f26100c0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48808/0 (socket says 172.21.15.148:48808) 2023-10-30T12:18:04.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.102+0000 7f261d4c8640 1 -- 172.21.15.148:0/2456017482 learned_addr learned my addr 172.21.15.148:0/2456017482 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.102+0000 7f261d4c8640 10 monclient(hunting): get_auth_request con 0x7f2610034d70 auth_method 0 2023-10-30T12:18:04.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.102+0000 7f261d4c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.102+0000 7f261d4c8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.102+0000 7f261d4c8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.103+0000 7f05793d0640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2023-10-30T12:18:04.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.103+0000 7f05793d0640 1 -- 172.21.15.148:0/2860908784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05741451d0 msgr2=0x7f057413b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.103+0000 7f05793d0640 1 --2- 172.21.15.148:0/2860908784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05741451d0 0x7f057413b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.103+0000 7f05793d0640 1 -- 172.21.15.148:0/2860908784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f05740721b0 msgr2=0x7f05740b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.103+0000 7f05793d0640 1 --2- 172.21.15.148:0/2860908784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f05740721b0 0x7f05740b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.103+0000 7f05793d0640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.103+0000 7f05793d0640 1 monclient: found mon.noname-c 2023-10-30T12:18:04.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.103+0000 7f05793d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.103+0000 7f05793d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.103+0000 7f05793d0640 1 -- 172.21.15.148:0/2860908784 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05740b4a10 con 0x7f057406bb80 2023-10-30T12:18:04.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.104+0000 7f8c6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.104+0000 7f8c6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.104+0000 7f8c6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.105+0000 7f261d4c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.105+0000 7f261d4c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.105+0000 7f261d4c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.105+0000 7f05793d0640 1 --2- 172.21.15.148:0/2860908784 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f057406bb80 0x7f05740b4410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f056400b4a0 tx=0x7f056400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.105+0000 7f0569ffb640 10 client.?.objecter ms_handle_connect 0x7f057406bb80 2023-10-30T12:18:04.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.105+0000 7f0569ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:04.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.106+0000 7f261ccc7640 1 --2- 172.21.15.148:0/2456017482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26100c3c80 0x7f26100c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.106+0000 7f261ccc7640 10 monclient(hunting): get_auth_request con 0x7f26100c3c80 auth_method 0 2023-10-30T12:18:04.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.106+0000 7f261ccc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.106+0000 7f261ccc7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.106+0000 7f261ccc7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.106+0000 7f8c6359e640 1 --2- 172.21.15.148:0/323870612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c5c0cc7b0 0x7f8c5c18fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261ccc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261ccc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261ccc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261d4c8640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2023-10-30T12:18:04.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261d4c8640 1 -- 172.21.15.148:0/2456017482 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 msgr2=0x7f26100bbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261d4c8640 1 --2- 172.21.15.148:0/2456017482 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 0x7f26100bbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261d4c8640 1 -- 172.21.15.148:0/2456017482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26100c3c80 msgr2=0x7f26100c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261d4c8640 1 --2- 172.21.15.148:0/2456017482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26100c3c80 0x7f26100c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261d4c8640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261d4c8640 1 monclient: found mon.noname-b 2023-10-30T12:18:04.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261d4c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261d4c8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.107+0000 7f261d4c8640 1 -- 172.21.15.148:0/2456017482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26100c9140 con 0x7f2610034d70 2023-10-30T12:18:04.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.108+0000 7f8c6359e640 10 monclient(hunting): get_auth_request con 0x7f8c5c0cc7b0 auth_method 0 2023-10-30T12:18:04.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.108+0000 7f8c6359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.108+0000 7f8c6359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.108+0000 7f8c6359e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.110+0000 7f8c6259c640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2023-10-30T12:18:04.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.110+0000 7f8c6259c640 1 -- 172.21.15.148:0/323870612 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8c5c02ac00 msgr2=0x7f8c5c030970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.110+0000 7f8c6259c640 1 --2- 172.21.15.148:0/323870612 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8c5c02ac00 0x7f8c5c030970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.110+0000 7f8c6259c640 1 -- 172.21.15.148:0/323870612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c5c0cc7b0 msgr2=0x7f8c5c18fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.110+0000 7f8c6259c640 1 --2- 172.21.15.148:0/323870612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c5c0cc7b0 0x7f8c5c18fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.110+0000 7f8c6259c640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.110+0000 7f8c6259c640 1 monclient: found mon.noname-b 2023-10-30T12:18:04.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.110+0000 7f8c6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.110+0000 7f8c6259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.110+0000 7f8c6259c640 1 -- 172.21.15.148:0/323870612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c02ff20 con 0x7f8c5c034d70 2023-10-30T12:18:04.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.111+0000 7f579affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57940354f0 0x7f579415cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60006/0 (socket says 172.21.15.148:60006) 2023-10-30T12:18:04.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.111+0000 7f579affd640 1 -- 172.21.15.148:0/1361266679 learned_addr learned my addr 172.21.15.148:0/1361266679 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.111+0000 7f579affd640 10 monclient(hunting): get_auth_request con 0x7f57940354f0 auth_method 0 2023-10-30T12:18:04.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.111+0000 7f579affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.111+0000 7f579affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.111+0000 7f579affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 1 -- 172.21.15.148:0/2860908784 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564010610 con 0x7f057406bb80 2023-10-30T12:18:04.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:04.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 10 monclient: dump: 2023-10-30T12:18:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.293 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.293 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.293 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.293 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.293 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 1 -- 172.21.15.148:0/2860908784 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0564010780 con 0x7f057406bb80 2023-10-30T12:18:04.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 1 -- 172.21.15.148:0/2860908784 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564010b40 con 0x7f057406bb80 2023-10-30T12:18:04.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:04.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f0569ffb640 10 monclient: dump: 2023-10-30T12:18:04.295 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.296 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.296 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.296 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.296 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.296 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.296 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.297 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.297 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.297 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f056b7fe640 4 set_mon_vals no callback set 2023-10-30T12:18:04.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f057b634640 5 monclient: authenticate success, global_id 4283 2023-10-30T12:18:04.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f057b634640 10 monclient: _renew_subs 2023-10-30T12:18:04.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f057b634640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.112+0000 7f057b634640 1 -- 172.21.15.148:0/2860908784 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05740ac430 con 0x7f057406bb80 2023-10-30T12:18:04.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.113+0000 7f057b634640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:04.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.113+0000 7f057b634640 10 monclient: _renew_subs 2023-10-30T12:18:04.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.113+0000 7f057b634640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.113+0000 7f057b634640 1 -- 172.21.15.148:0/2860908784 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057413bc00 con 0x7f057406bb80 2023-10-30T12:18:04.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.113+0000 7f057b634640 1 librados: init done 2023-10-30T12:18:04.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f057b634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f057b634640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f057b634640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f057b634640 1 -- 172.21.15.148:0/2860908784 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0574075550 con 0x7f057406bb80 2023-10-30T12:18:04.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 1 -- 172.21.15.148:0/910375795 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc00317c0 con 0x7f8dd00b7c10 2023-10-30T12:18:04.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:04.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 10 monclient: dump: 2023-10-30T12:18:04.310 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.310 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.311 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.311 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.312 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.313 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.313 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.313 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.314 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.314 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 1 -- 172.21.15.148:0/910375795 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc002e070 con 0x7f8dd00b7c10 2023-10-30T12:18:04.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dd4826640 5 monclient: authenticate success, global_id 4345 2023-10-30T12:18:04.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dd4826640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:04.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dd4826640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:04.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 1 -- 172.21.15.148:0/910375795 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0031b50 con 0x7f8dd00b7c10 2023-10-30T12:18:04.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dd4826640 4 set_mon_vals no callback set 2023-10-30T12:18:04.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:04.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.114+0000 7f8dcd59a640 10 monclient: dump: 2023-10-30T12:18:04.323 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.326 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.327 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.327 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.328 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.329 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.329 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.329 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.330 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.330 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f579affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f579affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f579affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8dd4826640 10 monclient: shutdown 2023-10-30T12:18:04.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8dd4826640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8dd4826640 1 -- 172.21.15.148:0/910375795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd00b7c10 msgr2=0x7f8dd00a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8dd4826640 1 --2- 172.21.15.148:0/910375795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd00b7c10 0x7f8dd00a8aa0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009870 tx=0x7f8dc002b860 comp rx=0 tx=0).stop 2023-10-30T12:18:04.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8dd4826640 1 -- 172.21.15.148:0/910375795 shutdown_connections 2023-10-30T12:18:04.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8dd4826640 1 --2- 172.21.15.148:0/910375795 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8dd013e730 0x7f8dd013eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8dd4826640 1 --2- 172.21.15.148:0/910375795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd00b7c10 0x7f8dd00a8aa0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8dd4826640 1 -- 172.21.15.148:0/910375795 >> 172.21.15.148:0/910375795 conn(0x7f8dd006c660 msgr2=0x7f8dd00b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8dd4826640 1 -- 172.21.15.148:0/910375795 shutdown_connections 2023-10-30T12:18:04.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8c6259c640 1 --2- 172.21.15.148:0/323870612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c034d70 0x7f8c5c18f950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8c58000c00 tx=0x7f8c580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8c4b7fe640 10 client.?.objecter ms_handle_connect 0x7f8c5c034d70 2023-10-30T12:18:04.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8c4b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:04.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.115+0000 7f8dd4826640 1 -- 172.21.15.148:0/910375795 wait complete. 2023-10-30T12:18:04.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.116+0000 7f261d4c8640 1 --2- 172.21.15.148:0/2456017482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2610034d70 0x7f26100c0640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f261400c450 tx=0x7f261400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.116+0000 7f260dffb640 10 client.?.objecter ms_handle_connect 0x7f2610034d70 2023-10-30T12:18:04.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.116+0000 7f260dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:04.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.116+0000 7f8dd4826640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.116+0000 7f8dd4826640 10 monclient: monmap: 2023-10-30T12:18:04.338 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.338 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:04.339 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:04.339 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:04.339 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.339 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:04.339 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:04.340 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:04.340 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:04.340 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 1 librados: starting msgr at 2023-10-30T12:18:04.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 1 librados: starting objecter 2023-10-30T12:18:04.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 1 Processor -- start 2023-10-30T12:18:04.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 1 -- start start 2023-10-30T12:18:04.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 1 librados: setting wanted keys 2023-10-30T12:18:04.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 1 librados: calling monclient init 2023-10-30T12:18:04.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 10 monclient: init 2023-10-30T12:18:04.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:18:04.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd00b7c10 0x7f8dd013b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 10 monclient(hunting): picked mon.noname-b con 0x7f8dd00b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:04.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f8dd4826640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dd013e730 0x7f8dd01da6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.117+0000 7f579bfff640 1 --2- 172.21.15.148:0/1361266679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57940c1b20 0x7f579415d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 10 monclient(hunting): picked mon.noname-a con 0x7f8dd013e730 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:04.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8dd0141940 0x7f8dd01daf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 10 monclient(hunting): picked mon.noname-c con 0x7f8dd0141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00b30b0 con 0x7f8dd013e730 2023-10-30T12:18:04.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd013b120 con 0x7f8dd00b7c10 2023-10-30T12:18:04.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd01cd9c0 con 0x7f8dd0141940 2023-10-30T12:18:04.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f8dd4826640 10 monclient(hunting): authenticate will time out at 1576.323136s 2023-10-30T12:18:04.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f579affd640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2023-10-30T12:18:04.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f579affd640 1 -- 172.21.15.148:0/1361266679 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f579402b730 msgr2=0x7f57940b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f579affd640 1 --2- 172.21.15.148:0/1361266679 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f579402b730 0x7f57940b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f579affd640 1 -- 172.21.15.148:0/1361266679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57940c1b20 msgr2=0x7f579415d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f579affd640 1 --2- 172.21.15.148:0/1361266679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57940c1b20 0x7f579415d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f579affd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f579affd640 1 monclient: found mon.noname-a 2023-10-30T12:18:04.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f579affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.118+0000 7f579affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.119+0000 7f579affd640 1 -- 172.21.15.148:0/1361266679 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57940b4960 con 0x7f57940354f0 2023-10-30T12:18:04.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.121+0000 7f579affd640 1 --2- 172.21.15.148:0/1361266679 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57940354f0 0x7f579415cee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f579c073730 tx=0x7f579c067180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.121+0000 7f5787fff640 10 client.?.objecter ms_handle_connect 0x7f57940354f0 2023-10-30T12:18:04.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.121+0000 7f5787fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:04.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 1 -- 172.21.15.148:0/323870612 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5803ad20 con 0x7f8c5c034d70 2023-10-30T12:18:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 10 monclient: dump: 2023-10-30T12:18:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.352 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.352 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.352 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.352 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.352 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.352 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.353 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.353 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 1 -- 172.21.15.148:0/323870612 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5803e030 con 0x7f8c5c034d70 2023-10-30T12:18:04.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 1 -- 172.21.15.148:0/323870612 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5803ae90 con 0x7f8c5c034d70 2023-10-30T12:18:04.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:04.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c4b7fe640 10 monclient: dump: 2023-10-30T12:18:04.355 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.355 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.355 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.355 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.355 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.356 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.356 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.356 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.356 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.356 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c68850640 5 monclient: authenticate success, global_id 4348 2023-10-30T12:18:04.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c68850640 10 monclient: _renew_subs 2023-10-30T12:18:04.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c68850640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c68850640 1 -- 172.21.15.148:0/323870612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5c02f2e0 con 0x7f8c5c034d70 2023-10-30T12:18:04.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c68850640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:04.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c68850640 10 monclient: _renew_subs 2023-10-30T12:18:04.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c68850640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c68850640 1 -- 172.21.15.148:0/323870612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5c0312b0 con 0x7f8c5c034d70 2023-10-30T12:18:04.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c68850640 1 librados: init done 2023-10-30T12:18:04.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.123+0000 7f8c6159a640 4 set_mon_vals no callback set 2023-10-30T12:18:04.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.124+0000 7f8c68850640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.124+0000 7f8c68850640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.124+0000 7f8c68850640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.124+0000 7f8c68850640 1 -- 172.21.15.148:0/323870612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c44005040 con 0x7f8c5c034d70 2023-10-30T12:18:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.125+0000 7f8dcdd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dd013e730 0x7f8dd01da6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.125+0000 7f8dce59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd00b7c10 0x7f8dd013b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.126+0000 7f8dcdd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dd013e730 0x7f8dd01da6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60022/0 (socket says 172.21.15.148:60022) 2023-10-30T12:18:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.126+0000 7f8dcdd9b640 1 -- 172.21.15.148:0/1214304707 learned_addr learned my addr 172.21.15.148:0/1214304707 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.126+0000 7f8dcdd9b640 10 monclient(hunting): get_auth_request con 0x7f8dd013e730 auth_method 0 2023-10-30T12:18:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.126+0000 7f8dcdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.126+0000 7f8dcdd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.126+0000 7f8dcdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.126+0000 7f8dcdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.126+0000 7f8dcdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.126+0000 7f8dcdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.129+0000 7f8dcdd9b640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2023-10-30T12:18:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.129+0000 7f8dcdd9b640 1 -- 172.21.15.148:0/1214304707 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8dd0141940 msgr2=0x7f8dd01daf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.129+0000 7f8dcdd9b640 1 --2- 172.21.15.148:0/1214304707 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8dd0141940 0x7f8dd01daf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.129+0000 7f8dcdd9b640 1 -- 172.21.15.148:0/1214304707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd00b7c10 msgr2=0x7f8dd013b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.129+0000 7f8dcdd9b640 1 --2- 172.21.15.148:0/1214304707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd00b7c10 0x7f8dd013b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.129+0000 7f8dcdd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.129+0000 7f8dcdd9b640 1 monclient: found mon.noname-a 2023-10-30T12:18:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.129+0000 7f8dcdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.129+0000 7f8dcdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.129+0000 7f8dcdd9b640 1 -- 172.21.15.148:0/1214304707 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd01cc930 con 0x7f8dd013e730 2023-10-30T12:18:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.132+0000 7f8dcdd9b640 1 --2- 172.21.15.148:0/1214304707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dd013e730 0x7f8dd01da6a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8dc400b410 tx=0x7f8dc400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.132+0000 7f8dbeffd640 10 client.?.objecter ms_handle_connect 0x7f8dd013e730 2023-10-30T12:18:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.132+0000 7f8dbeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.140+0000 7f0569ffb640 1 -- 172.21.15.148:0/2860908784 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f0564010cb0 con 0x7f057406bb80 2023-10-30T12:18:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.140+0000 7f0569ffb640 1 --2- 172.21.15.148:0/2860908784 >> v2:172.21.15.148:6800/3205090179 conn(0x7f055806c810 0x7f055806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.140+0000 7f0578bcf640 1 --2- 172.21.15.148:0/2860908784 >> v2:172.21.15.148:6800/3205090179 conn(0x7f055806c810 0x7f055806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.140+0000 7f0578bcf640 10 monclient: get_auth_request con 0x7f055806c810 auth_method 0 2023-10-30T12:18:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.141+0000 7f0578bcf640 1 --2- 172.21.15.148:0/2860908784 >> v2:172.21.15.148:6800/3205090179 conn(0x7f055806c810 0x7f055806ecd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f056c035000 tx=0x7f056c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.141+0000 7f0569ffb640 10 client.4283.objecter ms_handle_connect 0x7f055806c810 2023-10-30T12:18:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.146+0000 7fd664b33640 10 monclient: get_monmap_and_config 2023-10-30T12:18:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.146+0000 7fd664b33640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.146+0000 7fd664b33640 10 monclient: monmap: 2023-10-30T12:18:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:04.147100+0000 2023-10-30T12:18:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:04.147100+0000 2023-10-30T12:18:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:04.375 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.146+0000 7fd664b33640 1 Processor -- start 2023-10-30T12:18:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.146+0000 7fd664b33640 1 -- start start 2023-10-30T12:18:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.146+0000 7fd664b33640 10 monclient: init 2023-10-30T12:18:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-10-30T12:18:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6600b7c10 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd660141940 0x7fd660141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 10 monclient(hunting): picked mon.noname-c con 0x7fd660141940 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd66013e730 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6600a9350 con 0x7fd6600b7c10 2023-10-30T12:18:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6600aa6a0 con 0x7fd66013e730 2023-10-30T12:18:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6600670a0 con 0x7fd660141940 2023-10-30T12:18:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.147+0000 7fd664b33640 10 monclient(hunting): authenticate will time out at 1576.352474s 2023-10-30T12:18:04.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.148+0000 7fd65e59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd6600b7c10 0x7fd6600a8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53046/0 (socket says 172.21.15.148:53046) 2023-10-30T12:18:04.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.148+0000 7fd65e59c640 1 -- 172.21.15.148:0/2602505503 learned_addr learned my addr 172.21.15.148:0/2602505503 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:18:04.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 1 -- 172.21.15.148:0/2700389748 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c011830 con 0x7f802806bc50 2023-10-30T12:18:04.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:04.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 10 monclient: dump: 2023-10-30T12:18:04.385 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.385 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.385 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.386 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.386 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.386 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.387 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.387 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.388 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.388 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 1 -- 172.21.15.148:0/2700389748 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801c00d040 con 0x7f802806bc50 2023-10-30T12:18:04.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f802759e640 5 monclient: authenticate success, global_id 4314 2023-10-30T12:18:04.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f802759e640 10 monclient: _renew_subs 2023-10-30T12:18:04.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f802759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f802759e640 1 -- 172.21.15.148:0/2700389748 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802812d070 con 0x7f802806bc50 2023-10-30T12:18:04.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 1 -- 172.21.15.148:0/2700389748 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c011be0 con 0x7f802806bc50 2023-10-30T12:18:04.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:04.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8016ffd640 10 monclient: dump: 2023-10-30T12:18:04.393 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.393 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.393 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.394 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.394 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.395 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.395 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.395 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.396 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.396 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f8024d99640 4 set_mon_vals no callback set 2023-10-30T12:18:04.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f802759e640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:04.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f802759e640 10 monclient: _renew_subs 2023-10-30T12:18:04.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f802759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f802759e640 1 -- 172.21.15.148:0/2700389748 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802813fdb0 con 0x7f802806bc50 2023-10-30T12:18:04.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.293+0000 7f802759e640 1 librados: init done 2023-10-30T12:18:04.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.294+0000 7f802759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.294+0000 7f802759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.294+0000 7f802759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.294+0000 7f802759e640 1 -- 172.21.15.148:0/2700389748 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8028071b40 con 0x7f802806bc50 2023-10-30T12:18:04.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 1 -- 172.21.15.148:0/1214304707 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4011830 con 0x7f8dd013e730 2023-10-30T12:18:04.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:04.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 10 monclient: dump: 2023-10-30T12:18:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.405 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.405 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.405 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.406 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.406 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 1 -- 172.21.15.148:0/1214304707 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc400d040 con 0x7f8dd013e730 2023-10-30T12:18:04.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 1 -- 172.21.15.148:0/1214304707 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4011be0 con 0x7f8dd013e730 2023-10-30T12:18:04.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dd4826640 5 monclient: authenticate success, global_id 4329 2023-10-30T12:18:04.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dd4826640 10 monclient: _renew_subs 2023-10-30T12:18:04.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dd4826640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dd4826640 1 -- 172.21.15.148:0/1214304707 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd01cd570 con 0x7f8dd013e730 2023-10-30T12:18:04.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dccd99640 4 set_mon_vals no callback set 2023-10-30T12:18:04.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:04.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dbeffd640 10 monclient: dump: 2023-10-30T12:18:04.411 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.412 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.412 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.412 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.412 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.413 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.413 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.413 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.413 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.414 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dd4826640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:04.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dd4826640 10 monclient: _renew_subs 2023-10-30T12:18:04.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dd4826640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dd4826640 1 -- 172.21.15.148:0/1214304707 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd01cad30 con 0x7f8dd013e730 2023-10-30T12:18:04.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.299+0000 7f8dd4826640 1 librados: init done 2023-10-30T12:18:04.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7fd65d59a640 1 -- 172.21.15.148:0/2602505503 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489051649 0 0) 0x7fd650001110 con 0x7fd6600b7c10 2023-10-30T12:18:04.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7fd65d59a640 10 monclient(hunting): _negotiate 2023-10-30T12:18:04.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7fd65d59a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7fd65d59a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7fd65d59a640 10 monclient(hunting): my global_id is 4332 2023-10-30T12:18:04.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7fd65d59a640 1 -- 172.21.15.148:0/2602505503 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd648003520 con 0x7fd6600b7c10 2023-10-30T12:18:04.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 1 -- 172.21.15.148:0/1361266679 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c04f050 con 0x7f57940354f0 2023-10-30T12:18:04.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:04.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 10 monclient: dump: 2023-10-30T12:18:04.419 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.420 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.420 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.420 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.420 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.420 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.420 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.421 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.421 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.421 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 1 -- 172.21.15.148:0/1361266679 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579c072040 con 0x7f57940354f0 2023-10-30T12:18:04.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 1 -- 172.21.15.148:0/1361266679 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c0723d0 con 0x7f57940354f0 2023-10-30T12:18:04.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f57a1f33640 5 monclient: authenticate success, global_id 4326 2023-10-30T12:18:04.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f57a1f33640 10 monclient: _renew_subs 2023-10-30T12:18:04.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f57a1f33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f57a1f33640 1 -- 172.21.15.148:0/1361266679 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57940c4f40 con 0x7f57940354f0 2023-10-30T12:18:04.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5799ffb640 4 set_mon_vals no callback set 2023-10-30T12:18:04.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:04.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f5787fff640 10 monclient: dump: 2023-10-30T12:18:04.424 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.425 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.425 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.425 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.425 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.425 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.426 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.426 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.426 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.426 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f57a1f33640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:04.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f57a1f33640 10 monclient: _renew_subs 2023-10-30T12:18:04.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f57a1f33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f57a1f33640 1 -- 172.21.15.148:0/1361266679 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57940c43f0 con 0x7f57940354f0 2023-10-30T12:18:04.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.300+0000 7f57a1f33640 1 librados: init done 2023-10-30T12:18:04.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.301+0000 7f8dd4826640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.301+0000 7f8dd4826640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.301+0000 7f8dd4826640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.301+0000 7f8dd4826640 1 -- 172.21.15.148:0/1214304707 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd00aa9c0 con 0x7f8dd013e730 2023-10-30T12:18:04.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.301+0000 7f57a1f33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.301+0000 7f57a1f33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.301+0000 7f57a1f33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:04.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.301+0000 7f57a1f33640 1 -- 172.21.15.148:0/1361266679 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5794008970 con 0x7f57940354f0 2023-10-30T12:18:04.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 1 -- 172.21.15.148:0/2287172660 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8504031830 con 0x7f85100adb20 2023-10-30T12:18:04.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:04.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 10 monclient: dump: 2023-10-30T12:18:04.434 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.434 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.434 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.435 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.435 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.435 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.435 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.435 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.436 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.436 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 1 -- 172.21.15.148:0/2287172660 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850402d040 con 0x7f85100adb20 2023-10-30T12:18:04.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8517b7c640 5 monclient: authenticate success, global_id 4305 2023-10-30T12:18:04.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8517b7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:04.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8517b7c640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:04.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 1 -- 172.21.15.148:0/2287172660 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8504031bc0 con 0x7f85100adb20 2023-10-30T12:18:04.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8517b7c640 4 set_mon_vals no callback set 2023-10-30T12:18:04.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:04.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.302+0000 7f8514916640 10 monclient: dump: 2023-10-30T12:18:04.439 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.439 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.439 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.440 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.440 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.440 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.440 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.441 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.441 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.441 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.303+0000 7f8517b7c640 10 monclient: shutdown 2023-10-30T12:18:04.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.303+0000 7f8517b7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.303+0000 7f8517b7c640 1 -- 172.21.15.148:0/2287172660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85100adb20 msgr2=0x7f85100adf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.303+0000 7f8517b7c640 1 --2- 172.21.15.148:0/2287172660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85100adb20 0x7f85100adf50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f85040097a0 tx=0x7f850402b8d0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.303+0000 7f8517b7c640 1 -- 172.21.15.148:0/2287172660 shutdown_connections 2023-10-30T12:18:04.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.303+0000 7f8517b7c640 1 --2- 172.21.15.148:0/2287172660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85100adb20 0x7f85100adf50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.303+0000 7f8517b7c640 1 -- 172.21.15.148:0/2287172660 >> 172.21.15.148:0/2287172660 conn(0x7f8510069dc0 msgr2=0x7f851006a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.303+0000 7f8517b7c640 1 -- 172.21.15.148:0/2287172660 shutdown_connections 2023-10-30T12:18:04.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.303+0000 7f8517b7c640 1 -- 172.21.15.148:0/2287172660 wait complete. 2023-10-30T12:18:04.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.304+0000 7f8517b7c640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.304+0000 7f8517b7c640 10 monclient: monmap: 2023-10-30T12:18:04.445 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.446 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:04.446 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:04.446 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:04.447 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.447 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:04.447 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:04.448 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:04.448 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:04.448 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.304+0000 7f8517b7c640 1 librados: starting msgr at 2023-10-30T12:18:04.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.304+0000 7f8517b7c640 1 librados: starting objecter 2023-10-30T12:18:04.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.304+0000 7f8517b7c640 1 Processor -- start 2023-10-30T12:18:04.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.304+0000 7f8517b7c640 1 -- start start 2023-10-30T12:18:04.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.304+0000 7f8517b7c640 1 librados: setting wanted keys 2023-10-30T12:18:04.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.304+0000 7f8517b7c640 1 librados: calling monclient init 2023-10-30T12:18:04.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.304+0000 7f8517b7c640 10 monclient: init 2023-10-30T12:18:04.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:04.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f85100a9750 0x7f85101d8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 10 monclient(hunting): picked mon.noname-c con 0x7f85100a9750 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85100adb20 0x7f85101c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 10 monclient(hunting): picked mon.noname-a con 0x7f85100adb20 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:04.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100ae490 0x7f85101d4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 10 monclient(hunting): picked mon.noname-b con 0x7f85100ae490 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:04.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85100aca80 con 0x7f85100adb20 2023-10-30T12:18:04.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85101d8fe0 con 0x7f85100ae490 2023-10-30T12:18:04.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85101d8580 con 0x7f85100a9750 2023-10-30T12:18:04.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.305+0000 7f8517b7c640 10 monclient(hunting): authenticate will time out at 1576.510657s 2023-10-30T12:18:04.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.361+0000 7f8c4b7fe640 1 -- 172.21.15.148:0/323870612 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f8c5804d450 con 0x7f8c5c034d70 2023-10-30T12:18:04.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.362+0000 7f8c4b7fe640 1 --2- 172.21.15.148:0/323870612 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8c5006c810 0x7f8c5006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.362+0000 7f8c62d9d640 1 --2- 172.21.15.148:0/323870612 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8c5006c810 0x7f8c5006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.362+0000 7f8c62d9d640 10 monclient: get_auth_request con 0x7f8c5006c810 auth_method 0 2023-10-30T12:18:04.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.362+0000 7f8c4b7fe640 10 client.4348.objecter ms_handle_connect 0x7f8c5006c810 2023-10-30T12:18:04.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.362+0000 7f8c62d9d640 1 --2- 172.21.15.148:0/323870612 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8c5006c810 0x7f8c5006ecd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8c54000f60 tx=0x7f8c540079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.366+0000 7f8c4b7fe640 1 -- 172.21.15.148:0/323870612 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f8c5802d040 con 0x7f8c5c034d70 2023-10-30T12:18:04.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.366+0000 7f8c4b7fe640 10 client.4348.objecter ms_dispatch 0x7f8c64000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:04.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.366+0000 7f8c4b7fe640 3 client.4348.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:04.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.366+0000 7f8c4b7fe640 3 client.4348.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:04.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.366+0000 7f8c4b7fe640 20 client.4348.objecter dump_active .. 0 homeless 2023-10-30T12:18:04.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.375+0000 7f8c4b7fe640 1 -- 172.21.15.148:0/323870612 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8c5807a990 con 0x7f8c5c034d70 2023-10-30T12:18:04.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.375+0000 7f8c4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.375+0000 7f8c4b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 1 -- 172.21.15.148:0/2456017482 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26140052d0 con 0x7f2610034d70 2023-10-30T12:18:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 10 monclient: dump: 2023-10-30T12:18:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.462 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 1 -- 172.21.15.148:0/2456017482 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261400e040 con 0x7f2610034d70 2023-10-30T12:18:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 1 -- 172.21.15.148:0/2456017482 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614012630 con 0x7f2610034d70 2023-10-30T12:18:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260dffb640 10 monclient: dump: 2023-10-30T12:18:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.466 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f260f7fe640 4 set_mon_vals no callback set 2023-10-30T12:18:04.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f261e4ca640 5 monclient: authenticate success, global_id 4351 2023-10-30T12:18:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f261e4ca640 10 monclient: _renew_subs 2023-10-30T12:18:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f261e4ca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f261e4ca640 1 -- 172.21.15.148:0/2456017482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26100c8850 con 0x7f2610034d70 2023-10-30T12:18:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f261e4ca640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f261e4ca640 10 monclient: _renew_subs 2023-10-30T12:18:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f261e4ca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f261e4ca640 1 -- 172.21.15.148:0/2456017482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26100bc710 con 0x7f2610034d70 2023-10-30T12:18:04.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.378+0000 7f261e4ca640 1 librados: init done 2023-10-30T12:18:04.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.380+0000 7f261e4ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.380+0000 7f261e4ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.380+0000 7f261e4ca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.380+0000 7f261e4ca640 1 -- 172.21.15.148:0/2456017482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26100cbe70 con 0x7f2610034d70 2023-10-30T12:18:04.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.382+0000 7f8516119640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100ae490 0x7f85101d4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.383+0000 7fd65dd9b640 1 --2- 172.21.15.148:0/2602505503 >> v2:172.21.15.150:3300/0 conn(0x7fd660141940 0x7fd660141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.383+0000 7f8516119640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100ae490 0x7f85101d4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48862/0 (socket says 172.21.15.148:48862) 2023-10-30T12:18:04.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.383+0000 7f8516119640 1 -- 172.21.15.148:0/4004695197 learned_addr learned my addr 172.21.15.148:0/4004695197 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.383+0000 7f8516119640 10 monclient(hunting): get_auth_request con 0x7f85100ae490 auth_method 0 2023-10-30T12:18:04.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.383+0000 7f8516119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.383+0000 7f8516119640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.383+0000 7f8516119640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.385+0000 7f0569ffb640 1 -- 172.21.15.148:0/2860908784 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f0564057f60 con 0x7f057406bb80 2023-10-30T12:18:04.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.385+0000 7f0569ffb640 10 client.4283.objecter ms_dispatch 0x7f0574000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:04.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.385+0000 7f0569ffb640 3 client.4283.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:04.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.385+0000 7f0569ffb640 3 client.4283.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:04.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.385+0000 7f0569ffb640 20 client.4283.objecter dump_active .. 0 homeless 2023-10-30T12:18:04.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.385+0000 7f0569ffb640 1 -- 172.21.15.148:0/2860908784 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f056400eea0 con 0x7f057406bb80 2023-10-30T12:18:04.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.385+0000 7f0569ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.385+0000 7f0569ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:04.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.386+0000 7f8515918640 1 --2- 172.21.15.148:0/4004695197 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f85100a9750 0x7f85101d8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.389+0000 7f8515918640 10 monclient(hunting): get_auth_request con 0x7f85100a9750 auth_method 0 2023-10-30T12:18:04.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.389+0000 7f8515918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.389+0000 7f8515918640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.389+0000 7f8515918640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.391+0000 7f8515918640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.391+0000 7f8515918640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.391+0000 7f8515918640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.394+0000 7f8515918640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2023-10-30T12:18:04.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.394+0000 7f8515918640 1 -- 172.21.15.148:0/4004695197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100ae490 msgr2=0x7f85101d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.394+0000 7f8515918640 1 --2- 172.21.15.148:0/4004695197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100ae490 0x7f85101d4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.394+0000 7f8515918640 1 -- 172.21.15.148:0/4004695197 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85100adb20 msgr2=0x7f85101c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.394+0000 7f8515918640 1 --2- 172.21.15.148:0/4004695197 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85100adb20 0x7f85101c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.394+0000 7f8515918640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.394+0000 7f8515918640 1 monclient: found mon.noname-c 2023-10-30T12:18:04.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.394+0000 7f8515918640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.394+0000 7f8515918640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.394+0000 7f8515918640 1 -- 172.21.15.148:0/4004695197 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85101c9ae0 con 0x7f85100a9750 2023-10-30T12:18:04.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.399+0000 7f8515918640 1 --2- 172.21.15.148:0/4004695197 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f85100a9750 0x7f85101d8aa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f850000c480 tx=0x7f850000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.399+0000 7f850e7fc640 10 client.?.objecter ms_handle_connect 0x7f85100a9750 2023-10-30T12:18:04.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.399+0000 7f850e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:04.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 1 -- 172.21.15.148:0/4004695197 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500007480 con 0x7f85100a9750 2023-10-30T12:18:04.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:04.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 10 monclient: dump: 2023-10-30T12:18:04.486 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.486 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.486 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.487 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.487 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.487 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.487 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.487 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.488 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.488 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 1 -- 172.21.15.148:0/4004695197 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850000e040 con 0x7f85100a9750 2023-10-30T12:18:04.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f8517b7c640 5 monclient: authenticate success, global_id 4286 2023-10-30T12:18:04.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f8517b7c640 10 monclient: _renew_subs 2023-10-30T12:18:04.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f8517b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f8517b7c640 1 -- 172.21.15.148:0/4004695197 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85101d50e0 con 0x7f85100a9750 2023-10-30T12:18:04.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 1 -- 172.21.15.148:0/4004695197 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85000059a0 con 0x7f85100a9750 2023-10-30T12:18:04.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:04.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850e7fc640 10 monclient: dump: 2023-10-30T12:18:04.490 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.491 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.491 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.491 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.491 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.491 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.491 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.492 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.492 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.492 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f8517b7c640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:04.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f850ffff640 4 set_mon_vals no callback set 2023-10-30T12:18:04.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f8517b7c640 10 monclient: _renew_subs 2023-10-30T12:18:04.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f8517b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f8517b7c640 1 -- 172.21.15.148:0/4004695197 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85101d8280 con 0x7f85100a9750 2023-10-30T12:18:04.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.404+0000 7f8517b7c640 1 librados: init done 2023-10-30T12:18:04.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.406+0000 7f8517b7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.406+0000 7f8517b7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.406+0000 7f8517b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.406+0000 7f8517b7c640 1 -- 172.21.15.148:0/4004695197 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851006ac00 con 0x7f85100a9750 2023-10-30T12:18:04.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.429+0000 7f850e7fc640 1 -- 172.21.15.148:0/4004695197 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f850001b3b0 con 0x7f85100a9750 2023-10-30T12:18:04.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.430+0000 7f850e7fc640 1 --2- 172.21.15.148:0/4004695197 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84fc06c7f0 0x7f84fc06ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.430+0000 7f8515117640 1 --2- 172.21.15.148:0/4004695197 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84fc06c7f0 0x7f84fc06ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.430+0000 7f8515117640 10 monclient: get_auth_request con 0x7f84fc06c7f0 auth_method 0 2023-10-30T12:18:04.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.431+0000 7f850e7fc640 10 client.4286.objecter ms_handle_connect 0x7f84fc06c7f0 2023-10-30T12:18:04.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.431+0000 7f8515117640 1 --2- 172.21.15.148:0/4004695197 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84fc06c7f0 0x7f84fc06ecb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8504036000 tx=0x7f8504030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.572+0000 7f057b634640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:18:04.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.572+0000 7f057b634640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:18:04.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.572+0000 7f057b634640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.572+0000 7f057b634640 1 -- 172.21.15.148:0/2860908784 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f05740ac430 con 0x7f057406bb80 2023-10-30T12:18:04.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.617+0000 7f8c68850640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:18:04.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.617+0000 7f8c68850640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:18:04.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.617+0000 7f8c68850640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.617+0000 7f8c68850640 1 -- 172.21.15.148:0/323870612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8c440055c0 con 0x7f8c5c034d70 2023-10-30T12:18:04.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.632+0000 7fd65dd9b640 10 monclient(hunting): get_auth_request con 0x7fd660141940 auth_method 0 2023-10-30T12:18:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.632+0000 7fd65dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.632+0000 7fd65dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.632+0000 7fd65dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.633+0000 7fd65dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.633+0000 7fd65dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.633+0000 7fd65dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.636+0000 7fd65dd9b640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2023-10-30T12:18:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.636+0000 7fd65dd9b640 1 -- 172.21.15.148:0/2602505503 >> v1:172.21.15.150:6789/0 conn(0x7fd66013e730 legacy=0x7fd66013eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.636+0000 7fd65dd9b640 1 -- 172.21.15.148:0/2602505503 >> v1:172.21.15.148:6789/0 conn(0x7fd6600b7c10 legacy=0x7fd6600a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.636+0000 7fd65dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.636+0000 7fd65dd9b640 1 monclient: found mon.noname-c 2023-10-30T12:18:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.636+0000 7fd65dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.636+0000 7fd65dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.636+0000 7fd65dd9b640 1 -- 172.21.15.148:0/2602505503 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6600aa910 con 0x7fd660141940 2023-10-30T12:18:04.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.639+0000 7fd65dd9b640 1 --2- 172.21.15.148:0/2602505503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd660141940 0x7fd660141d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd6540097a0 tx=0x7fd65402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4921341921432bd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.656+0000 7f260dffb640 1 -- 172.21.15.148:0/2456017482 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f26140127a0 con 0x7f2610034d70 2023-10-30T12:18:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.656+0000 7f260dffb640 1 --2- 172.21.15.148:0/2456017482 >> v2:172.21.15.148:6800/3205090179 conn(0x7f25fc06c810 0x7f25fc06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.656+0000 7f261ccc7640 1 --2- 172.21.15.148:0/2456017482 >> v2:172.21.15.148:6800/3205090179 conn(0x7f25fc06c810 0x7f25fc06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.656+0000 7f261ccc7640 10 monclient: get_auth_request con 0x7f25fc06c810 auth_method 0 2023-10-30T12:18:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.657+0000 7f261ccc7640 1 --2- 172.21.15.148:0/2456017482 >> v2:172.21.15.148:6800/3205090179 conn(0x7f25fc06c810 0x7f25fc06ecd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f26180731e0 tx=0x7f261806d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.657+0000 7f260dffb640 10 client.4351.objecter ms_handle_connect 0x7f25fc06c810 2023-10-30T12:18:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.658+0000 7f260dffb640 1 -- 172.21.15.148:0/2456017482 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f2614058a00 con 0x7f2610034d70 2023-10-30T12:18:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.658+0000 7f260dffb640 10 client.4351.objecter ms_dispatch 0x7f2618000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.658+0000 7f260dffb640 3 client.4351.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.658+0000 7f260dffb640 3 client.4351.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.658+0000 7f260dffb640 20 client.4351.objecter dump_active .. 0 homeless 2023-10-30T12:18:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.668+0000 7f260dffb640 1 -- 172.21.15.148:0/2456017482 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2614058300 con 0x7f2610034d70 2023-10-30T12:18:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.668+0000 7f260dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.668+0000 7f260dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 1 -- 172.21.15.148:0/2602505503 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd654031830 con 0x7fd660141940 2023-10-30T12:18:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 10 monclient: dump: 2023-10-30T12:18:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.674 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 1 -- 172.21.15.148:0/2602505503 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65402d040 con 0x7fd660141940 2023-10-30T12:18:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 1 -- 172.21.15.148:0/2602505503 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd654031bc0 con 0x7fd660141940 2023-10-30T12:18:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd664b33640 5 monclient: authenticate success, global_id 4354 2023-10-30T12:18:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd664b33640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd664b33640 4 set_mon_vals no callback set 2023-10-30T12:18:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.669+0000 7fd65d59a640 10 monclient: dump: 2023-10-30T12:18:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.678 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.670+0000 7fd664b33640 10 monclient: shutdown 2023-10-30T12:18:04.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.670+0000 7fd664b33640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.670+0000 7fd664b33640 1 -- 172.21.15.148:0/2602505503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd660141940 msgr2=0x7fd660141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.670+0000 7fd664b33640 1 --2- 172.21.15.148:0/2602505503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd660141940 0x7fd660141d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd6540097a0 tx=0x7fd65402b8d0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.670+0000 7fd664b33640 1 -- 172.21.15.148:0/2602505503 shutdown_connections 2023-10-30T12:18:04.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.670+0000 7fd664b33640 1 --2- 172.21.15.148:0/2602505503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd660141940 0x7fd660141d90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.670+0000 7fd664b33640 1 -- 172.21.15.148:0/2602505503 >> 172.21.15.148:0/2602505503 conn(0x7fd66006c660 msgr2=0x7fd6600b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.670+0000 7fd664b33640 1 -- 172.21.15.148:0/2602505503 shutdown_connections 2023-10-30T12:18:04.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.670+0000 7fd664b33640 1 -- 172.21.15.148:0/2602505503 wait complete. 2023-10-30T12:18:04.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.672+0000 7fd664b33640 10 monclient: build_initial_monmap 2023-10-30T12:18:04.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.672+0000 7fd664b33640 10 monclient: monmap: 2023-10-30T12:18:04.682 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:04.682 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:04.682 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:04.683 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:04.683 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:04.683 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:04.683 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:04.683 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:04.684 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:04.684 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.672+0000 7fd664b33640 1 librados: starting msgr at 2023-10-30T12:18:04.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.672+0000 7fd664b33640 1 librados: starting objecter 2023-10-30T12:18:04.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.672+0000 7fd664b33640 1 Processor -- start 2023-10-30T12:18:04.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.672+0000 7fd664b33640 1 -- start start 2023-10-30T12:18:04.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.672+0000 7fd664b33640 1 librados: setting wanted keys 2023-10-30T12:18:04.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.672+0000 7fd664b33640 1 librados: calling monclient init 2023-10-30T12:18:04.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.672+0000 7fd664b33640 10 monclient: init 2023-10-30T12:18:04.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:04.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:04.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6600b7c10 0x7fd6600afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 10 monclient(hunting): picked mon.noname-b con 0x7fd6600b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:04.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd66013e730 0x7fd6600b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 10 monclient(hunting): picked mon.noname-c con 0x7fd66013e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:04.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660141940 0x7fd6601383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:04.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 10 monclient(hunting): picked mon.noname-a con 0x7fd660141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:04.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66013eb60 con 0x7fd660141940 2023-10-30T12:18:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601c7ce0 con 0x7fd6600b7c10 2023-10-30T12:18:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601395e0 con 0x7fd66013e730 2023-10-30T12:18:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:04.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.673+0000 7fd664b33640 10 monclient(hunting): authenticate will time out at 1576.878577s 2023-10-30T12:18:04.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.677+0000 7f8c4b7fe640 1 -- 172.21.15.148:0/323870612 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8c5807a290 con 0x7f8c5c034d70 2023-10-30T12:18:04.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.677+0000 7f8c4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:18:04.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.677+0000 7f8c4b7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:04.691 INFO:teuthology.orchestra.run.smithi148.stdout:64424509444 2023-10-30T12:18:04.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 10 librados: watch_flush enter 2023-10-30T12:18:04.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 10 librados: watch_flush exit 2023-10-30T12:18:04.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 20 client.4348.objecter shutdown clearing up homeless session... 2023-10-30T12:18:04.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 10 client.4348.objecter successfully canceled tick 2023-10-30T12:18:04.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 -- 172.21.15.148:0/323870612 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8c5006c810 msgr2=0x7f8c5006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 --2- 172.21.15.148:0/323870612 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8c5006c810 0x7f8c5006ecd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8c54000f60 tx=0x7f8c540079b0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 10 monclient: shutdown 2023-10-30T12:18:04.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 -- 172.21.15.148:0/323870612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c034d70 msgr2=0x7f8c5c18f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 --2- 172.21.15.148:0/323870612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c034d70 0x7f8c5c18f950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8c58000c00 tx=0x7f8c580046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 -- 172.21.15.148:0/323870612 shutdown_connections 2023-10-30T12:18:04.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 --2- 172.21.15.148:0/323870612 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8c5006c810 0x7f8c5006ecd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 --2- 172.21.15.148:0/323870612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c5c0cc7b0 0x7f8c5c18fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 --2- 172.21.15.148:0/323870612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5c034d70 0x7f8c5c18f950 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 --2- 172.21.15.148:0/323870612 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8c5c02ac00 0x7f8c5c030970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 -- 172.21.15.148:0/323870612 >> 172.21.15.148:0/323870612 conn(0x7f8c5c0061c0 msgr2=0x7f8c5c0065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 -- 172.21.15.148:0/323870612 shutdown_connections 2023-10-30T12:18:04.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 -- 172.21.15.148:0/323870612 wait complete. 2023-10-30T12:18:04.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.681+0000 7f8c68850640 1 librados: shutdown 2023-10-30T12:18:04.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.688+0000 7fd65e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6600b7c10 0x7fd6600afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:04.698 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.5 2023-10-30T12:18:04.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.711+0000 7fd65e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6600b7c10 0x7fd6600afc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48880/0 (socket says 172.21.15.148:48880) 2023-10-30T12:18:04.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.711+0000 7fd65e59c640 1 -- 172.21.15.148:0/40752498 learned_addr learned my addr 172.21.15.148:0/40752498 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:04.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.711+0000 7fd65e59c640 10 monclient(hunting): get_auth_request con 0x7fd6600b7c10 auth_method 0 2023-10-30T12:18:04.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.711+0000 7fd65e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:04.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.711+0000 7fd65e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:04.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.711+0000 7fd65e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:04.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.713+0000 7fd65e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:04.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.713+0000 7fd65e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:04.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.713+0000 7fd65e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:04.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.716+0000 7fd65e59c640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2023-10-30T12:18:04.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.716+0000 7fd65e59c640 1 -- 172.21.15.148:0/40752498 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd66013e730 msgr2=0x7fd6600b0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.716+0000 7fd65e59c640 1 --2- 172.21.15.148:0/40752498 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd66013e730 0x7fd6600b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.716+0000 7fd65e59c640 1 -- 172.21.15.148:0/40752498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660141940 msgr2=0x7fd6601383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.716+0000 7fd65e59c640 1 --2- 172.21.15.148:0/40752498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660141940 0x7fd6601383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.716+0000 7fd65e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:18:04.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.716+0000 7fd65e59c640 1 monclient: found mon.noname-b 2023-10-30T12:18:04.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.716+0000 7fd65e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:04.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.716+0000 7fd65e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.716+0000 7fd65e59c640 1 -- 172.21.15.148:0/40752498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6600ae260 con 0x7fd6600b7c10 2023-10-30T12:18:04.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.719+0000 7fd65e59c640 1 --2- 172.21.15.148:0/40752498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6600b7c10 0x7fd6600afc10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd65000c730 tx=0x7fd65000cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:04.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.719+0000 7fd64effd640 10 client.?.objecter ms_handle_connect 0x7fd6600b7c10 2023-10-30T12:18:04.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.719+0000 7fd64effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:04.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.728+0000 7f850e7fc640 1 -- 172.21.15.148:0/4004695197 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f8500007630 con 0x7f85100a9750 2023-10-30T12:18:04.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.728+0000 7f850e7fc640 10 client.4286.objecter ms_dispatch 0x7f8510000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:04.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.728+0000 7f850e7fc640 3 client.4286.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:04.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.728+0000 7f850e7fc640 3 client.4286.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:04.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.728+0000 7f850e7fc640 20 client.4286.objecter dump_active .. 0 homeless 2023-10-30T12:18:04.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.728+0000 7f850e7fc640 1 -- 172.21.15.148:0/4004695197 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f85000576f0 con 0x7f85100a9750 2023-10-30T12:18:04.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.728+0000 7f850e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.728+0000 7f850e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:04.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.738+0000 7f0569ffb640 1 -- 172.21.15.148:0/2860908784 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0564014e70 con 0x7f057406bb80 2023-10-30T12:18:04.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.738+0000 7f0569ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:18:04.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.738+0000 7f0569ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:04.743 INFO:teuthology.orchestra.run.smithi148.stdout:64424509444 2023-10-30T12:18:04.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 10 librados: watch_flush enter 2023-10-30T12:18:04.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 10 librados: watch_flush exit 2023-10-30T12:18:04.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 20 client.4283.objecter shutdown clearing up homeless session... 2023-10-30T12:18:04.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 10 client.4283.objecter successfully canceled tick 2023-10-30T12:18:04.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 -- 172.21.15.148:0/2860908784 >> v2:172.21.15.148:6800/3205090179 conn(0x7f055806c810 msgr2=0x7f055806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 --2- 172.21.15.148:0/2860908784 >> v2:172.21.15.148:6800/3205090179 conn(0x7f055806c810 0x7f055806ecd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f056c035000 tx=0x7f056c030000 comp rx=0 tx=0).stop 2023-10-30T12:18:04.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 10 monclient: shutdown 2023-10-30T12:18:04.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 -- 172.21.15.148:0/2860908784 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f057406bb80 msgr2=0x7f05740b4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 --2- 172.21.15.148:0/2860908784 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f057406bb80 0x7f05740b4410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f056400b4a0 tx=0x7f056400b960 comp rx=0 tx=0).stop 2023-10-30T12:18:04.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 -- 172.21.15.148:0/2860908784 shutdown_connections 2023-10-30T12:18:04.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 --2- 172.21.15.148:0/2860908784 >> v2:172.21.15.148:6800/3205090179 conn(0x7f055806c810 0x7f055806ecd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 --2- 172.21.15.148:0/2860908784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05741451d0 0x7f057413b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 --2- 172.21.15.148:0/2860908784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f05740721b0 0x7f05740b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 --2- 172.21.15.148:0/2860908784 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f057406bb80 0x7f05740b4410 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 -- 172.21.15.148:0/2860908784 >> 172.21.15.148:0/2860908784 conn(0x7f05740b54e0 msgr2=0x7f05741cb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.741+0000 7f057b634640 1 -- 172.21.15.148:0/2860908784 shutdown_connections 2023-10-30T12:18:04.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.742+0000 7f057b634640 1 -- 172.21.15.148:0/2860908784 wait complete. 2023-10-30T12:18:04.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.742+0000 7f057b634640 1 librados: shutdown 2023-10-30T12:18:04.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.4 2023-10-30T12:18:04.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.856+0000 7f261e4ca640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:18:04.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.856+0000 7f261e4ca640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:18:04.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.856+0000 7f261e4ca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.856+0000 7f261e4ca640 1 -- 172.21.15.148:0/2456017482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f261002bf90 con 0x7f2610034d70 2023-10-30T12:18:04.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.915+0000 7f8517b7c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:18:04.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.915+0000 7f8517b7c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:18:04.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.915+0000 7f8517b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:04.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.915+0000 7f8517b7c640 1 -- 172.21.15.148:0/4004695197 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f85101d50e0 con 0x7f85100a9750 2023-10-30T12:18:04.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.919+0000 7f850e7fc640 1 -- 172.21.15.148:0/4004695197 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f850005b790 con 0x7f85100a9750 2023-10-30T12:18:04.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.919+0000 7f850e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:18:04.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.919+0000 7f850e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:04.924 INFO:teuthology.orchestra.run.smithi148.stdout:64424509444 2023-10-30T12:18:04.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 10 librados: watch_flush enter 2023-10-30T12:18:04.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 10 librados: watch_flush exit 2023-10-30T12:18:04.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 20 client.4286.objecter shutdown clearing up homeless session... 2023-10-30T12:18:04.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 10 client.4286.objecter successfully canceled tick 2023-10-30T12:18:04.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 -- 172.21.15.148:0/4004695197 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84fc06c7f0 msgr2=0x7f84fc06ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 --2- 172.21.15.148:0/4004695197 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84fc06c7f0 0x7f84fc06ecb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8504036000 tx=0x7f8504030000 comp rx=0 tx=0).stop 2023-10-30T12:18:04.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 10 monclient: shutdown 2023-10-30T12:18:04.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 -- 172.21.15.148:0/4004695197 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f85100a9750 msgr2=0x7f85101d8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 --2- 172.21.15.148:0/4004695197 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f85100a9750 0x7f85101d8aa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f850000c480 tx=0x7f850000c940 comp rx=0 tx=0).stop 2023-10-30T12:18:04.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 -- 172.21.15.148:0/4004695197 shutdown_connections 2023-10-30T12:18:04.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 --2- 172.21.15.148:0/4004695197 >> v2:172.21.15.148:6800/3205090179 conn(0x7f84fc06c7f0 0x7f84fc06ecb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 --2- 172.21.15.148:0/4004695197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100ae490 0x7f85101d4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 --2- 172.21.15.148:0/4004695197 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85100adb20 0x7f85101c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 --2- 172.21.15.148:0/4004695197 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f85100a9750 0x7f85101d8aa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 -- 172.21.15.148:0/4004695197 >> 172.21.15.148:0/4004695197 conn(0x7f8510069dc0 msgr2=0x7f851013dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.921+0000 7f8517b7c640 1 -- 172.21.15.148:0/4004695197 shutdown_connections 2023-10-30T12:18:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.922+0000 7f8517b7c640 1 -- 172.21.15.148:0/4004695197 wait complete. 2023-10-30T12:18:04.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.922+0000 7f8517b7c640 1 librados: shutdown 2023-10-30T12:18:04.934 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.2 2023-10-30T12:18:04.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.981+0000 7f260dffb640 1 -- 172.21.15.148:0/2456017482 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f261405c450 con 0x7f2610034d70 2023-10-30T12:18:04.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.981+0000 7f260dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:18:04.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.981+0000 7f260dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:04.985 INFO:teuthology.orchestra.run.smithi148.stdout:64424509443 2023-10-30T12:18:04.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 1 -- 172.21.15.148:0/40752498 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd650005ab0 con 0x7fd6600b7c10 2023-10-30T12:18:04.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:04.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 10 monclient: dump: 2023-10-30T12:18:04.986 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.987 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.987 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.987 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.987 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.988 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.988 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.988 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.988 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.988 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 10 monclient: _finish_auth 0 2023-10-30T12:18:04.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:04.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:04.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 1 -- 172.21.15.148:0/40752498 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65000e040 con 0x7fd6600b7c10 2023-10-30T12:18:04.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:04.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 1 -- 172.21.15.148:0/40752498 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd650005e60 con 0x7fd6600b7c10 2023-10-30T12:18:04.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd664b33640 5 monclient: authenticate success, global_id 4363 2023-10-30T12:18:04.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd664b33640 10 monclient: _renew_subs 2023-10-30T12:18:04.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd664b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd664b33640 1 -- 172.21.15.148:0/40752498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6601388f0 con 0x7fd6600b7c10 2023-10-30T12:18:04.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd65cd99640 4 set_mon_vals no callback set 2023-10-30T12:18:04.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:04.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:04.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.982+0000 7fd64effd640 10 monclient: dump: 2023-10-30T12:18:04.991 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:04.991 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:04.992 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.992 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:04.992 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:04.992 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:04.992 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:04.993 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:04.993 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:04.993 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:04.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.983+0000 7fd664b33640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:04.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.983+0000 7fd664b33640 10 monclient: _renew_subs 2023-10-30T12:18:04.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.983+0000 7fd664b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.983+0000 7fd664b33640 1 -- 172.21.15.148:0/40752498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd660138d90 con 0x7fd6600b7c10 2023-10-30T12:18:04.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.983+0000 7fd664b33640 1 librados: init done 2023-10-30T12:18:04.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 10 librados: watch_flush enter 2023-10-30T12:18:04.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 10 librados: watch_flush exit 2023-10-30T12:18:04.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 20 client.4351.objecter shutdown clearing up homeless session... 2023-10-30T12:18:04.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 10 client.4351.objecter successfully canceled tick 2023-10-30T12:18:04.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7fd664b33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7fd664b33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:04.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7fd664b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:04.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7fd664b33640 1 -- 172.21.15.148:0/40752498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6600aeea0 con 0x7fd6600b7c10 2023-10-30T12:18:04.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 -- 172.21.15.148:0/2456017482 >> v2:172.21.15.148:6800/3205090179 conn(0x7f25fc06c810 msgr2=0x7f25fc06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 --2- 172.21.15.148:0/2456017482 >> v2:172.21.15.148:6800/3205090179 conn(0x7f25fc06c810 0x7f25fc06ecd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f26180731e0 tx=0x7f261806d040 comp rx=0 tx=0).stop 2023-10-30T12:18:04.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 10 monclient: shutdown 2023-10-30T12:18:04.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:04.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 -- 172.21.15.148:0/2456017482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2610034d70 msgr2=0x7f26100c0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:04.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 --2- 172.21.15.148:0/2456017482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2610034d70 0x7f26100c0640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f261400c450 tx=0x7f261400c910 comp rx=0 tx=0).stop 2023-10-30T12:18:04.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 -- 172.21.15.148:0/2456017482 shutdown_connections 2023-10-30T12:18:04.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 --2- 172.21.15.148:0/2456017482 >> v2:172.21.15.148:6800/3205090179 conn(0x7f25fc06c810 0x7f25fc06ecd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:04.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 --2- 172.21.15.148:0/2456017482 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f26100cc580 0x7f26100bbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 --2- 172.21.15.148:0/2456017482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26100c3c80 0x7f26100c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 --2- 172.21.15.148:0/2456017482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2610034d70 0x7f26100c0640 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 -- 172.21.15.148:0/2456017482 >> 172.21.15.148:0/2456017482 conn(0x7f26100061c0 msgr2=0x7f26100c3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:05.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 -- 172.21.15.148:0/2456017482 shutdown_connections 2023-10-30T12:18:05.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 -- 172.21.15.148:0/2456017482 wait complete. 2023-10-30T12:18:05.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:04.984+0000 7f261e4ca640 1 librados: shutdown 2023-10-30T12:18:05.001 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509443 for osd.0 2023-10-30T12:18:05.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.060+0000 7f8dbeffd640 1 -- 172.21.15.148:0/1214304707 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f8dc4011d50 con 0x7f8dd013e730 2023-10-30T12:18:05.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.060+0000 7f8dbeffd640 1 --2- 172.21.15.148:0/1214304707 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8db006c810 0x7f8db006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.060+0000 7f8dce59c640 1 --2- 172.21.15.148:0/1214304707 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8db006c810 0x7f8db006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.060+0000 7f8dce59c640 10 monclient: get_auth_request con 0x7f8db006c810 auth_method 0 2023-10-30T12:18:05.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.061+0000 7f8dce59c640 1 --2- 172.21.15.148:0/1214304707 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8db006c810 0x7f8db006ecd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0036000 tx=0x7f8dc003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:05.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.061+0000 7f8dbeffd640 10 client.4329.objecter ms_handle_connect 0x7f8db006c810 2023-10-30T12:18:05.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.072+0000 7f8016ffd640 1 -- 172.21.15.148:0/2700389748 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f801c011d50 con 0x7f802806bc50 2023-10-30T12:18:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.072+0000 7f8016ffd640 1 --2- 172.21.15.148:0/2700389748 >> v2:172.21.15.148:6800/3205090179 conn(0x7f800806c7c0 0x7f800806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.072+0000 7f802659c640 1 --2- 172.21.15.148:0/2700389748 >> v2:172.21.15.148:6800/3205090179 conn(0x7f800806c7c0 0x7f800806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.072+0000 7f802659c640 10 monclient: get_auth_request con 0x7f800806c7c0 auth_method 0 2023-10-30T12:18:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.072+0000 7f8016ffd640 10 client.4314.objecter ms_handle_connect 0x7f800806c7c0 2023-10-30T12:18:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.072+0000 7f802659c640 1 --2- 172.21.15.148:0/2700389748 >> v2:172.21.15.148:6800/3205090179 conn(0x7f800806c7c0 0x7f800806ec80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8018036000 tx=0x7f801803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.094+0000 7f8016ffd640 1 -- 172.21.15.148:0/2700389748 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f801c057f20 con 0x7f802806bc50 2023-10-30T12:18:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.094+0000 7f8016ffd640 10 client.4314.objecter ms_dispatch 0x7f8028000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.094+0000 7f8016ffd640 3 client.4314.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.094+0000 7f8016ffd640 3 client.4314.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.094+0000 7f8016ffd640 20 client.4314.objecter dump_active .. 0 homeless 2023-10-30T12:18:05.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.094+0000 7f8016ffd640 1 -- 172.21.15.148:0/2700389748 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f801c05bfc0 con 0x7f802806bc50 2023-10-30T12:18:05.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.094+0000 7f8016ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:05.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.094+0000 7f8016ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:05.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.101+0000 7f8dbeffd640 1 -- 172.21.15.148:0/1214304707 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f8dc4058b50 con 0x7f8dd013e730 2023-10-30T12:18:05.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.101+0000 7f8dbeffd640 10 client.4329.objecter ms_dispatch 0x7f8dd0000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:05.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.101+0000 7f8dbeffd640 3 client.4329.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:05.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.101+0000 7f8dbeffd640 3 client.4329.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:05.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.101+0000 7f8dbeffd640 20 client.4329.objecter dump_active .. 0 homeless 2023-10-30T12:18:05.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.108+0000 7f8dbeffd640 1 -- 172.21.15.148:0/1214304707 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8dc4058450 con 0x7f8dd013e730 2023-10-30T12:18:05.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.108+0000 7f8dbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:05.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.108+0000 7f8dbeffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:05.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.122+0000 7f5787fff640 1 -- 172.21.15.148:0/1361266679 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f579c09d3f0 con 0x7f57940354f0 2023-10-30T12:18:05.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.123+0000 7f5787fff640 1 --2- 172.21.15.148:0/1361266679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f578006c7f0 0x7f578006ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.123+0000 7f579b7fe640 1 --2- 172.21.15.148:0/1361266679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f578006c7f0 0x7f578006ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.123+0000 7f579b7fe640 10 monclient: get_auth_request con 0x7f578006c7f0 auth_method 0 2023-10-30T12:18:05.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.123+0000 7f579b7fe640 1 --2- 172.21.15.148:0/1361266679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f578006c7f0 0x7f578006ecb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57880015d0 tx=0x7f57880078b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:05.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.123+0000 7f5787fff640 10 client.4326.objecter ms_handle_connect 0x7f578006c7f0 2023-10-30T12:18:05.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.138+0000 7f5787fff640 1 -- 172.21.15.148:0/1361266679 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f579c0d2160 con 0x7f57940354f0 2023-10-30T12:18:05.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.138+0000 7f5787fff640 10 client.4326.objecter ms_dispatch 0x7f579c000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:05.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.138+0000 7f5787fff640 3 client.4326.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:05.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.138+0000 7f5787fff640 3 client.4326.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:05.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.138+0000 7f5787fff640 20 client.4326.objecter dump_active .. 0 homeless 2023-10-30T12:18:05.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.138+0000 7f5787fff640 1 -- 172.21.15.148:0/1361266679 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f57940c43f0 con 0x7f57940354f0 2023-10-30T12:18:05.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.138+0000 7f5787fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:05.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.138+0000 7f5787fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:05.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.247+0000 7fd64effd640 1 -- 172.21.15.148:0/40752498 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fd650007070 con 0x7fd6600b7c10 2023-10-30T12:18:05.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.248+0000 7fd64effd640 1 --2- 172.21.15.148:0/40752498 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd64006c810 0x7fd64006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.248+0000 7fd65dd9b640 1 --2- 172.21.15.148:0/40752498 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd64006c810 0x7fd64006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.248+0000 7fd65dd9b640 10 monclient: get_auth_request con 0x7fd64006c810 auth_method 0 2023-10-30T12:18:05.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.248+0000 7fd65dd9b640 1 --2- 172.21.15.148:0/40752498 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd64006c810 0x7fd64006ecd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd654036000 tx=0x7fd654030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:05.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.248+0000 7fd64effd640 10 client.4363.objecter ms_handle_connect 0x7fd64006c810 2023-10-30T12:18:05.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.250+0000 7fd64effd640 1 -- 172.21.15.148:0/40752498 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fd6500576f0 con 0x7fd6600b7c10 2023-10-30T12:18:05.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.250+0000 7fd64effd640 10 client.4363.objecter ms_dispatch 0x7fd660000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:05.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.250+0000 7fd64effd640 3 client.4363.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:05.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.250+0000 7fd64effd640 3 client.4363.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:05.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.251+0000 7fd64effd640 20 client.4363.objecter dump_active .. 0 homeless 2023-10-30T12:18:05.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.260+0000 7fd64effd640 1 -- 172.21.15.148:0/40752498 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd650007da0 con 0x7fd6600b7c10 2023-10-30T12:18:05.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.260+0000 7fd64effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:05.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.260+0000 7fd64effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:05.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.299+0000 7f802759e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:18:05.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.299+0000 7f802759e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:18:05.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.299+0000 7f802759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:05.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.299+0000 7f802759e640 1 -- 172.21.15.148:0/2700389748 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f802812d070 con 0x7f802806bc50 2023-10-30T12:18:05.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.340+0000 7f57a1f33640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:18:05.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.340+0000 7f57a1f33640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:18:05.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.340+0000 7f57a1f33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:05.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.340+0000 7f57a1f33640 1 -- 172.21.15.148:0/1361266679 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5794005790 con 0x7f57940354f0 2023-10-30T12:18:05.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.355+0000 7f8dd4826640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:18:05.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.355+0000 7f8dd4826640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:18:05.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.355+0000 7f8dd4826640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:05.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.355+0000 7f8dd4826640 1 -- 172.21.15.148:0/1214304707 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8dd00a97c0 con 0x7f8dd013e730 2023-10-30T12:18:05.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.406+0000 7f8dbeffd640 1 -- 172.21.15.148:0/1214304707 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8dc405c5a0 con 0x7f8dd013e730 2023-10-30T12:18:05.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.406+0000 7f8dbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:18:05.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.406+0000 7f8dbeffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:05.408 INFO:teuthology.orchestra.run.smithi148.stdout:64424509444 2023-10-30T12:18:05.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.408+0000 7f8dd4826640 10 librados: watch_flush enter 2023-10-30T12:18:05.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 10 librados: watch_flush exit 2023-10-30T12:18:05.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 20 client.4329.objecter shutdown clearing up homeless session... 2023-10-30T12:18:05.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 10 client.4329.objecter successfully canceled tick 2023-10-30T12:18:05.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 -- 172.21.15.148:0/1214304707 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8db006c810 msgr2=0x7f8db006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 --2- 172.21.15.148:0/1214304707 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8db006c810 0x7f8db006ecd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0036000 tx=0x7f8dc003d040 comp rx=0 tx=0).stop 2023-10-30T12:18:05.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 10 monclient: shutdown 2023-10-30T12:18:05.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:05.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 -- 172.21.15.148:0/1214304707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dd013e730 msgr2=0x7f8dd01da6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 --2- 172.21.15.148:0/1214304707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dd013e730 0x7f8dd01da6a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8dc400b410 tx=0x7f8dc400b8d0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 -- 172.21.15.148:0/1214304707 shutdown_connections 2023-10-30T12:18:05.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 --2- 172.21.15.148:0/1214304707 >> v2:172.21.15.148:6800/3205090179 conn(0x7f8db006c810 0x7f8db006ecd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 --2- 172.21.15.148:0/1214304707 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8dd0141940 0x7f8dd01daf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 --2- 172.21.15.148:0/1214304707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dd013e730 0x7f8dd01da6a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 --2- 172.21.15.148:0/1214304707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd00b7c10 0x7f8dd013b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 -- 172.21.15.148:0/1214304707 >> 172.21.15.148:0/1214304707 conn(0x7f8dd006c660 msgr2=0x7f8dd013f840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:05.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 -- 172.21.15.148:0/1214304707 shutdown_connections 2023-10-30T12:18:05.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 -- 172.21.15.148:0/1214304707 wait complete. 2023-10-30T12:18:05.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.409+0000 7f8dd4826640 1 librados: shutdown 2023-10-30T12:18:05.426 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.1 2023-10-30T12:18:05.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.477+0000 7fd664b33640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:18:05.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.477+0000 7fd664b33640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:18:05.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.477+0000 7fd664b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:05.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.477+0000 7fd664b33640 1 -- 172.21.15.148:0/40752498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd6601388f0 con 0x7fd6600b7c10 2023-10-30T12:18:05.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.481+0000 7fd64effd640 1 -- 172.21.15.148:0/40752498 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd65005b790 con 0x7fd6600b7c10 2023-10-30T12:18:05.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.481+0000 7fd64effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:18:05.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.481+0000 7fd64effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:05.483 INFO:teuthology.orchestra.run.smithi148.stdout:64424509445 2023-10-30T12:18:05.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.483+0000 7fd664b33640 10 librados: watch_flush enter 2023-10-30T12:18:05.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 10 librados: watch_flush exit 2023-10-30T12:18:05.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 20 client.4363.objecter shutdown clearing up homeless session... 2023-10-30T12:18:05.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 10 client.4363.objecter successfully canceled tick 2023-10-30T12:18:05.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 -- 172.21.15.148:0/40752498 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd64006c810 msgr2=0x7fd64006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 --2- 172.21.15.148:0/40752498 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd64006c810 0x7fd64006ecd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd654036000 tx=0x7fd654030000 comp rx=0 tx=0).stop 2023-10-30T12:18:05.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 10 monclient: shutdown 2023-10-30T12:18:05.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:05.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 -- 172.21.15.148:0/40752498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6600b7c10 msgr2=0x7fd6600afc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 --2- 172.21.15.148:0/40752498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6600b7c10 0x7fd6600afc10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd65000c730 tx=0x7fd65000cbf0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 -- 172.21.15.148:0/40752498 shutdown_connections 2023-10-30T12:18:05.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 --2- 172.21.15.148:0/40752498 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd64006c810 0x7fd64006ecd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 --2- 172.21.15.148:0/40752498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660141940 0x7fd6601383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 --2- 172.21.15.148:0/40752498 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd66013e730 0x7fd6600b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 --2- 172.21.15.148:0/40752498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6600b7c10 0x7fd6600afc10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 -- 172.21.15.148:0/40752498 >> 172.21.15.148:0/40752498 conn(0x7fd66006c660 msgr2=0x7fd66013aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 -- 172.21.15.148:0/40752498 shutdown_connections 2023-10-30T12:18:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 -- 172.21.15.148:0/40752498 wait complete. 2023-10-30T12:18:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.484+0000 7fd664b33640 1 librados: shutdown 2023-10-30T12:18:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.491+0000 7f8016ffd640 1 -- 172.21.15.148:0/2700389748 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f801c060170 con 0x7f802806bc50 2023-10-30T12:18:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.491+0000 7f8016ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:18:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.491+0000 7f8016ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:05.493 INFO:teuthology.orchestra.run.smithi148.stdout:64424509444 2023-10-30T12:18:05.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 10 librados: watch_flush enter 2023-10-30T12:18:05.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 10 librados: watch_flush exit 2023-10-30T12:18:05.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 20 client.4314.objecter shutdown clearing up homeless session... 2023-10-30T12:18:05.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 10 client.4314.objecter successfully canceled tick 2023-10-30T12:18:05.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 -- 172.21.15.148:0/2700389748 >> v2:172.21.15.148:6800/3205090179 conn(0x7f800806c7c0 msgr2=0x7f800806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 --2- 172.21.15.148:0/2700389748 >> v2:172.21.15.148:6800/3205090179 conn(0x7f800806c7c0 0x7f800806ec80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8018036000 tx=0x7f801803c040 comp rx=0 tx=0).stop 2023-10-30T12:18:05.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 10 monclient: shutdown 2023-10-30T12:18:05.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:05.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 -- 172.21.15.148:0/2700389748 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f802806bc50 msgr2=0x7f802812b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 --2- 172.21.15.148:0/2700389748 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f802806bc50 0x7f802812b470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f801c00b3f0 tx=0x7f801c00b8b0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 -- 172.21.15.148:0/2700389748 shutdown_connections 2023-10-30T12:18:05.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 --2- 172.21.15.148:0/2700389748 >> v2:172.21.15.148:6800/3205090179 conn(0x7f800806c7c0 0x7f800806ec80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 --2- 172.21.15.148:0/2700389748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8028140d10 0x7f802812b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 --2- 172.21.15.148:0/2700389748 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f802806bc50 0x7f802812b470 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 --2- 172.21.15.148:0/2700389748 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f802806ac90 0x7f802813f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 -- 172.21.15.148:0/2700389748 >> 172.21.15.148:0/2700389748 conn(0x7f8028069dc0 msgr2=0x7f802812fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:05.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.494+0000 7f802759e640 1 -- 172.21.15.148:0/2700389748 shutdown_connections 2023-10-30T12:18:05.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.495+0000 7f802759e640 1 -- 172.21.15.148:0/2700389748 wait complete. 2023-10-30T12:18:05.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.495+0000 7f802759e640 1 librados: shutdown 2023-10-30T12:18:05.501 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.7 2023-10-30T12:18:05.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.499+0000 7f5787fff640 1 -- 172.21.15.148:0/1361266679 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f579c0d1a60 con 0x7f57940354f0 2023-10-30T12:18:05.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.499+0000 7f5787fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:18:05.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.499+0000 7f5787fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:05.505 INFO:teuthology.orchestra.run.smithi148.stdout:55834574853 2023-10-30T12:18:05.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.502+0000 7f57a1f33640 10 librados: watch_flush enter 2023-10-30T12:18:05.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.502+0000 7f57a1f33640 10 librados: watch_flush exit 2023-10-30T12:18:05.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.502+0000 7f57a1f33640 20 client.4326.objecter shutdown clearing up homeless session... 2023-10-30T12:18:05.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.502+0000 7f57a1f33640 10 client.4326.objecter successfully canceled tick 2023-10-30T12:18:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.502+0000 7f57a1f33640 1 -- 172.21.15.148:0/1361266679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f578006c7f0 msgr2=0x7f578006ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.502+0000 7f57a1f33640 1 --2- 172.21.15.148:0/1361266679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f578006c7f0 0x7f578006ecb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57880015d0 tx=0x7f57880078b0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.502+0000 7f57a1f33640 10 monclient: shutdown 2023-10-30T12:18:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.502+0000 7f57a1f33640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.502+0000 7f57a1f33640 1 -- 172.21.15.148:0/1361266679 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57940354f0 msgr2=0x7f579415cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.502+0000 7f57a1f33640 1 --2- 172.21.15.148:0/1361266679 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57940354f0 0x7f579415cee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f579c073730 tx=0x7f579c067180 comp rx=0 tx=0).stop 2023-10-30T12:18:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.503+0000 7f57a1f33640 1 -- 172.21.15.148:0/1361266679 shutdown_connections 2023-10-30T12:18:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.503+0000 7f57a1f33640 1 --2- 172.21.15.148:0/1361266679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f578006c7f0 0x7f578006ecb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.503+0000 7f57a1f33640 1 --2- 172.21.15.148:0/1361266679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57940c1b20 0x7f579415d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.503+0000 7f57a1f33640 1 --2- 172.21.15.148:0/1361266679 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57940354f0 0x7f579415cee0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.503+0000 7f57a1f33640 1 --2- 172.21.15.148:0/1361266679 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f579402b730 0x7f57940b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.503+0000 7f57a1f33640 1 -- 172.21.15.148:0/1361266679 >> 172.21.15.148:0/1361266679 conn(0x7f5794006a70 msgr2=0x7f57940be750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.503+0000 7f57a1f33640 1 -- 172.21.15.148:0/1361266679 shutdown_connections 2023-10-30T12:18:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.503+0000 7f57a1f33640 1 -- 172.21.15.148:0/1361266679 wait complete. 2023-10-30T12:18:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.503+0000 7f57a1f33640 1 librados: shutdown 2023-10-30T12:18:05.511 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.3 2023-10-30T12:18:05.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2023-10-30T12:18:05.699 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:18:05.767 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:18:05.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.805+0000 7fcce63f0640 10 monclient: get_monmap_and_config 2023-10-30T12:18:05.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.805+0000 7fcce63f0640 10 monclient: build_initial_monmap 2023-10-30T12:18:05.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.805+0000 7fcce63f0640 10 monclient: monmap: 2023-10-30T12:18:05.807 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:05.807 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:05.808 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:05.806396+0000 2023-10-30T12:18:05.808 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:05.806396+0000 2023-10-30T12:18:05.808 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:05.808 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:05.809 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:05.809 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:05.809 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:05.809 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:05.810 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:05.810 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 1 Processor -- start 2023-10-30T12:18:05.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 1 -- start start 2023-10-30T12:18:05.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 10 monclient: init 2023-10-30T12:18:05.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:05.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-10-30T12:18:05.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcce00b7c10 0x7fcce00a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 10 monclient(hunting): picked mon.noname-c con 0x7fcce00b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:05.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcce0141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:05.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce013e730 0x7fcce013eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 10 monclient(hunting): picked mon.noname-b con 0x7fcce013e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:05.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce00a9350 con 0x7fcce0141940 2023-10-30T12:18:05.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:05.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce00670a0 con 0x7fcce00b7c10 2023-10-30T12:18:05.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:05.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00b7ff0 con 0x7fcce013e730 2023-10-30T12:18:05.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:05.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.806+0000 7fcce63f0640 10 monclient(hunting): authenticate will time out at 1578.011678s 2023-10-30T12:18:05.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.807+0000 7fcce498d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce013e730 0x7fcce013eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.807+0000 7fcce498d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce013e730 0x7fcce013eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54596/0 (socket says 172.21.15.148:54596) 2023-10-30T12:18:05.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.807+0000 7fcce498d640 1 -- 172.21.15.148:0/966333478 learned_addr learned my addr 172.21.15.148:0/966333478 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:05.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.807+0000 7fcce498d640 10 monclient(hunting): get_auth_request con 0x7fcce013e730 auth_method 0 2023-10-30T12:18:05.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.807+0000 7fcce498d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:05.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.807+0000 7fcce498d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:05.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.807+0000 7fcce498d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:05.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.808+0000 7fcce498d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:05.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.808+0000 7fcce498d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:05.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.808+0000 7fcce498d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:05.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.808+0000 7fccdffff640 1 --2- 172.21.15.148:0/966333478 >> v2:172.21.15.150:3300/0 conn(0x7fcce00b7c10 0x7fcce00a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.809+0000 7fccdffff640 10 monclient(hunting): get_auth_request con 0x7fcce00b7c10 auth_method 0 2023-10-30T12:18:05.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.809+0000 7fccdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:05.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.809+0000 7fccdffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:05.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.809+0000 7fccdffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:05.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.810+0000 7fcce498d640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2023-10-30T12:18:05.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.810+0000 7fcce498d640 1 -- 172.21.15.148:0/966333478 >> v2:172.21.15.150:3300/0 conn(0x7fcce00b7c10 msgr2=0x7fcce00a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:05.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.810+0000 7fcce498d640 1 --2- 172.21.15.148:0/966333478 >> v2:172.21.15.150:3300/0 conn(0x7fcce00b7c10 0x7fcce00a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.810+0000 7fcce498d640 1 -- 172.21.15.148:0/966333478 >> v1:172.21.15.150:6789/0 conn(0x7fcce0141940 legacy=0x7fcce0141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:05.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.810+0000 7fcce498d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:05.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.810+0000 7fcce498d640 1 monclient: found mon.noname-b 2023-10-30T12:18:05.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.810+0000 7fcce498d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:05.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.810+0000 7fcce498d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:05.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.810+0000 7fcce498d640 1 -- 172.21.15.148:0/966333478 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce00aa6a0 con 0x7fcce013e730 2023-10-30T12:18:05.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.813+0000 7fcce498d640 1 --2- 172.21.15.148:0/966333478 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce013e730 0x7fcce013eb80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009e50 tx=0x7fccd802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=626a88037666c8b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:05.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 1 -- 172.21.15.148:0/966333478 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd802c9e0 con 0x7fcce013e730 2023-10-30T12:18:05.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:05.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:05.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 10 monclient: dump: 2023-10-30T12:18:05.825 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:05.826 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:05.826 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.826 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.826 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:05.826 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:05.827 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:05.827 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:05.827 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:05.827 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:05.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:05.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:05.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 1 -- 172.21.15.148:0/966333478 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd802cb50 con 0x7fcce013e730 2023-10-30T12:18:05.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:05.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 1 -- 172.21.15.148:0/966333478 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd80343f0 con 0x7fcce013e730 2023-10-30T12:18:05.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fcce63f0640 5 monclient: authenticate success, global_id 4292 2023-10-30T12:18:05.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fcce63f0640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:05.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fcce63f0640 4 set_mon_vals no callback set 2023-10-30T12:18:05.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:05.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:05.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.817+0000 7fccdeffd640 10 monclient: dump: 2023-10-30T12:18:05.830 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:05.830 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:05.830 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.830 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.830 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:05.831 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:05.831 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:05.831 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:05.831 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:05.831 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.818+0000 7fcce63f0640 10 monclient: shutdown 2023-10-30T12:18:05.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.818+0000 7fcce63f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:05.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.818+0000 7fcce63f0640 1 -- 172.21.15.148:0/966333478 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce013e730 msgr2=0x7fcce013eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.818+0000 7fcce63f0640 1 --2- 172.21.15.148:0/966333478 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce013e730 0x7fcce013eb80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009e50 tx=0x7fccd802be40 comp rx=0 tx=0).stop 2023-10-30T12:18:05.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.818+0000 7fcce63f0640 1 -- 172.21.15.148:0/966333478 shutdown_connections 2023-10-30T12:18:05.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.818+0000 7fcce63f0640 1 --2- 172.21.15.148:0/966333478 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce013e730 0x7fcce013eb80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.818+0000 7fcce63f0640 1 --2- 172.21.15.148:0/966333478 >> v2:172.21.15.150:3300/0 conn(0x7fcce00b7c10 0x7fcce00a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.818+0000 7fcce63f0640 1 -- 172.21.15.148:0/966333478 >> 172.21.15.148:0/966333478 conn(0x7fcce006c660 msgr2=0x7fcce00b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:05.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.818+0000 7fcce63f0640 1 -- 172.21.15.148:0/966333478 shutdown_connections 2023-10-30T12:18:05.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.818+0000 7fcce63f0640 1 -- 172.21.15.148:0/966333478 wait complete. 2023-10-30T12:18:05.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.819+0000 7fcce63f0640 10 monclient: build_initial_monmap 2023-10-30T12:18:05.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.819+0000 7fcce63f0640 10 monclient: monmap: 2023-10-30T12:18:05.834 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:05.834 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:05.834 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:05.835 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:05.835 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:05.835 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:05.835 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:05.835 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:05.836 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:05.836 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.819+0000 7fcce63f0640 1 librados: starting msgr at 2023-10-30T12:18:05.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.819+0000 7fcce63f0640 1 librados: starting objecter 2023-10-30T12:18:05.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.819+0000 7fcce63f0640 1 Processor -- start 2023-10-30T12:18:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.820+0000 7fcce63f0640 1 -- start start 2023-10-30T12:18:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.820+0000 7fcce63f0640 1 librados: setting wanted keys 2023-10-30T12:18:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.820+0000 7fcce63f0640 1 librados: calling monclient init 2023-10-30T12:18:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.820+0000 7fcce63f0640 10 monclient: init 2023-10-30T12:18:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:18:05.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcce00b7c10 0x7fcce00aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 10 monclient(hunting): picked mon.noname-b con 0x7fcce00b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:05.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcce013e730 0x7fcce01ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 10 monclient(hunting): picked mon.noname-a con 0x7fcce013e730 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:05.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce0141940 0x7fcce01ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 10 monclient(hunting): picked mon.noname-c con 0x7fcce0141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:05.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:05.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00b30b0 con 0x7fcce013e730 2023-10-30T12:18:05.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:05.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce0138420 con 0x7fcce00b7c10 2023-10-30T12:18:05.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:05.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00b0ef0 con 0x7fcce0141940 2023-10-30T12:18:05.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:05.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce63f0640 10 monclient(hunting): authenticate will time out at 1578.025999s 2023-10-30T12:18:05.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fccdf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcce013e730 0x7fcce01ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fcce498d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce0141940 0x7fcce01ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fccdffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcce00b7c10 0x7fcce00aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fccdf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcce013e730 0x7fcce01ce020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60040/0 (socket says 172.21.15.148:60040) 2023-10-30T12:18:05.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fccdf7fe640 1 -- 172.21.15.148:0/3608585370 learned_addr learned my addr 172.21.15.148:0/3608585370 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:05.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fccdf7fe640 10 monclient(hunting): get_auth_request con 0x7fcce013e730 auth_method 0 2023-10-30T12:18:05.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.821+0000 7fccdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:05.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:05.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:05.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdffff640 10 monclient(hunting): get_auth_request con 0x7fcce00b7c10 auth_method 0 2023-10-30T12:18:05.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fcce498d640 10 monclient(hunting): get_auth_request con 0x7fcce0141940 auth_method 0 2023-10-30T12:18:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fcce498d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fcce498d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fcce498d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:05.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:05.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:05.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:05.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:05.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:05.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fccdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:05.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fcce498d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:05.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fcce498d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:05.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.822+0000 7fcce498d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdf7fe640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2023-10-30T12:18:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdf7fe640 1 -- 172.21.15.148:0/3608585370 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce0141940 msgr2=0x7fcce01ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdf7fe640 1 --2- 172.21.15.148:0/3608585370 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce0141940 0x7fcce01ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdf7fe640 1 -- 172.21.15.148:0/3608585370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcce00b7c10 msgr2=0x7fcce00aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdf7fe640 1 --2- 172.21.15.148:0/3608585370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcce00b7c10 0x7fcce00aeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:18:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdf7fe640 1 monclient: found mon.noname-a 2023-10-30T12:18:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:05.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdf7fe640 1 -- 172.21.15.148:0/3608585370 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce00afd00 con 0x7fcce013e730 2023-10-30T12:18:05.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.825+0000 7fccdffff640 1 --2- 172.21.15.148:0/3608585370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcce00b7c10 0x7fcce00aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:05.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.828+0000 7fccdf7fe640 1 --2- 172.21.15.148:0/3608585370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcce013e730 0x7fcce01ce020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccd400b420 tx=0x7fccd400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:05.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.828+0000 7fccdcff9640 10 client.?.objecter ms_handle_connect 0x7fcce013e730 2023-10-30T12:18:05.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.828+0000 7fccdcff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:05.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 1 -- 172.21.15.148:0/3608585370 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4011830 con 0x7fcce013e730 2023-10-30T12:18:05.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:05.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:05.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 10 monclient: dump: 2023-10-30T12:18:05.850 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:05.851 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:05.851 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.851 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.851 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:05.851 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:05.852 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:05.852 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:05.852 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:05.852 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 10 monclient: _finish_auth 0 2023-10-30T12:18:05.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 10 monclient: _check_auth_tickets 2023-10-30T12:18:05.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:05.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 1 -- 172.21.15.148:0/3608585370 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd400d040 con 0x7fcce013e730 2023-10-30T12:18:05.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:05.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 1 -- 172.21.15.148:0/3608585370 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4011be0 con 0x7fcce013e730 2023-10-30T12:18:05.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:05.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:05.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccdcff9640 10 monclient: dump: 2023-10-30T12:18:05.854 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:05.854 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:05.854 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.855 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.855 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:05.855 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:05.855 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:05.855 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:05.856 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:05.856 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fcce63f0640 5 monclient: authenticate success, global_id 4335 2023-10-30T12:18:05.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fcce63f0640 10 monclient: _renew_subs 2023-10-30T12:18:05.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fcce63f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:05.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fcce63f0640 1 -- 172.21.15.148:0/3608585370 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce00adc50 con 0x7fcce013e730 2023-10-30T12:18:05.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fccde7fc640 4 set_mon_vals no callback set 2023-10-30T12:18:05.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fcce63f0640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:05.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fcce63f0640 10 monclient: _renew_subs 2023-10-30T12:18:05.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fcce63f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:05.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fcce63f0640 1 -- 172.21.15.148:0/3608585370 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce00af3c0 con 0x7fcce013e730 2023-10-30T12:18:05.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.832+0000 7fcce63f0640 1 librados: init done 2023-10-30T12:18:05.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.833+0000 7fcce63f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:05.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.833+0000 7fcce63f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:05.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.833+0000 7fcce63f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:05.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.833+0000 7fcce63f0640 1 -- 172.21.15.148:0/3608585370 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce00af650 con 0x7fcce013e730 2023-10-30T12:18:05.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.862+0000 7fccdcff9640 1 -- 172.21.15.148:0/3608585370 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fccd4011d50 con 0x7fcce013e730 2023-10-30T12:18:05.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.863+0000 7fccdcff9640 1 --2- 172.21.15.148:0/3608585370 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcccc06c7c0 0x7fcccc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.863+0000 7fccdffff640 1 --2- 172.21.15.148:0/3608585370 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcccc06c7c0 0x7fcccc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.863+0000 7fccdffff640 10 monclient: get_auth_request con 0x7fcccc06c7c0 auth_method 0 2023-10-30T12:18:05.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.863+0000 7fccdcff9640 10 client.4335.objecter ms_handle_connect 0x7fcccc06c7c0 2023-10-30T12:18:05.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.863+0000 7fccdffff640 1 --2- 172.21.15.148:0/3608585370 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcccc06c7c0 0x7fcccc06ec80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fccd0006fd0 tx=0x7fccd0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:05.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.891+0000 7efe2be56640 10 monclient: get_monmap_and_config 2023-10-30T12:18:05.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.891+0000 7efe2be56640 10 monclient: build_initial_monmap 2023-10-30T12:18:05.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.891+0000 7efe2be56640 10 monclient: monmap: 2023-10-30T12:18:05.893 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:05.894 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:05.894 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:05.892405+0000 2023-10-30T12:18:05.894 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:05.892405+0000 2023-10-30T12:18:05.894 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:05.895 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:05.895 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:05.895 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:05.895 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:05.895 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:05.896 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:05.896 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.891+0000 7efe2be56640 1 Processor -- start 2023-10-30T12:18:05.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 1 -- start start 2023-10-30T12:18:05.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 10 monclient: init 2023-10-30T12:18:05.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:05.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-10-30T12:18:05.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efe240b7c10 0x7efe240a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 10 monclient(hunting): picked mon.noname-c con 0x7efe240b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:05.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe24141940 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:05.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe2413e730 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:05.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe240a9350 con 0x7efe24141940 2023-10-30T12:18:05.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe240aa6a0 con 0x7efe2413e730 2023-10-30T12:18:05.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:05.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe240670a0 con 0x7efe240b7c10 2023-10-30T12:18:05.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:05.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.892+0000 7efe2be56640 10 monclient(hunting): authenticate will time out at 1578.097690s 2023-10-30T12:18:05.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.893+0000 7efe29bf2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efe240b7c10 0x7efe240a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.893+0000 7efe29bf2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efe240b7c10 0x7efe240a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48900/0 (socket says 172.21.15.148:48900) 2023-10-30T12:18:05.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.893+0000 7efe29bf2640 1 -- 172.21.15.148:0/3692285875 learned_addr learned my addr 172.21.15.148:0/3692285875 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:05.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.893+0000 7efe29bf2640 10 monclient(hunting): get_auth_request con 0x7efe240b7c10 auth_method 0 2023-10-30T12:18:05.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.893+0000 7efe29bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:05.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.893+0000 7efe29bf2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:05.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.893+0000 7efe29bf2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:05.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.894+0000 7efe29bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:05.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.894+0000 7efe29bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:05.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.894+0000 7efe29bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:05.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.896+0000 7efe29bf2640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2023-10-30T12:18:05.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.896+0000 7efe29bf2640 1 -- 172.21.15.148:0/3692285875 >> v1:172.21.15.150:6789/0 conn(0x7efe2413e730 legacy=0x7efe2413eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.896+0000 7efe29bf2640 1 -- 172.21.15.148:0/3692285875 >> v1:172.21.15.148:6789/0 conn(0x7efe24141940 legacy=0x7efe24141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:05.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.896+0000 7efe29bf2640 10 monclient: _finish_hunting 0 2023-10-30T12:18:05.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.896+0000 7efe29bf2640 1 monclient: found mon.noname-c 2023-10-30T12:18:05.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.896+0000 7efe29bf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:05.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.896+0000 7efe29bf2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:05.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.896+0000 7efe29bf2640 1 -- 172.21.15.148:0/3692285875 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe240aa910 con 0x7efe240b7c10 2023-10-30T12:18:05.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.900+0000 7efe29bf2640 1 --2- 172.21.15.148:0/3692285875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe240b7c10 0x7efe240a8aa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efe20009870 tx=0x7efe2002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ebfc4e6e5337dbe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:05.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.902+0000 7efe28bf0640 1 -- 172.21.15.148:0/3692285875 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20031870 con 0x7efe240b7c10 2023-10-30T12:18:05.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.902+0000 7efe28bf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:05.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.902+0000 7efe28bf0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:05.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.902+0000 7efe28bf0640 10 monclient: dump: 2023-10-30T12:18:05.908 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:05.908 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:05.909 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.909 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.909 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:05.909 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:05.909 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:05.910 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:05.910 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:05.910 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.902+0000 7efe28bf0640 10 monclient: _finish_auth 0 2023-10-30T12:18:05.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.902+0000 7efe28bf0640 10 monclient: _check_auth_tickets 2023-10-30T12:18:05.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.902+0000 7efe28bf0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:05.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.902+0000 7efe2be56640 5 monclient: authenticate success, global_id 4372 2023-10-30T12:18:05.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.902+0000 7efe2be56640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:05.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.904+0000 7efe28bf0640 1 -- 172.21.15.148:0/3692285875 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe200319e0 con 0x7efe240b7c10 2023-10-30T12:18:05.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.904+0000 7efe28bf0640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:05.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.904+0000 7efe28bf0640 1 -- 172.21.15.148:0/3692285875 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20031d80 con 0x7efe240b7c10 2023-10-30T12:18:05.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.904+0000 7efe28bf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:05.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.904+0000 7efe28bf0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:05.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.904+0000 7efe28bf0640 10 monclient: dump: 2023-10-30T12:18:05.913 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:05.913 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:05.914 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.914 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.914 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:05.914 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:05.915 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:05.915 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:05.915 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:05.915 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.904+0000 7efe2be56640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:05.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.904+0000 7efe2be56640 4 set_mon_vals no callback set 2023-10-30T12:18:05.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.905+0000 7efe2be56640 10 monclient: shutdown 2023-10-30T12:18:05.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.905+0000 7efe2be56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:05.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.905+0000 7efe2be56640 1 -- 172.21.15.148:0/3692285875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe240b7c10 msgr2=0x7efe240a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.905+0000 7efe2be56640 1 --2- 172.21.15.148:0/3692285875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe240b7c10 0x7efe240a8aa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efe20009870 tx=0x7efe2002b910 comp rx=0 tx=0).stop 2023-10-30T12:18:05.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.905+0000 7efe2be56640 1 -- 172.21.15.148:0/3692285875 shutdown_connections 2023-10-30T12:18:05.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.905+0000 7efe2be56640 1 --2- 172.21.15.148:0/3692285875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe240b7c10 0x7efe240a8aa0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.905+0000 7efe2be56640 1 -- 172.21.15.148:0/3692285875 >> 172.21.15.148:0/3692285875 conn(0x7efe2406c660 msgr2=0x7efe240b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:05.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.905+0000 7efe2be56640 1 -- 172.21.15.148:0/3692285875 shutdown_connections 2023-10-30T12:18:05.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.905+0000 7efe2be56640 1 -- 172.21.15.148:0/3692285875 wait complete. 2023-10-30T12:18:05.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.906+0000 7efe2be56640 10 monclient: build_initial_monmap 2023-10-30T12:18:05.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.906+0000 7efe2be56640 10 monclient: monmap: 2023-10-30T12:18:05.927 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:05.927 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:05.927 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:05.930 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.906+0000 7efe2be56640 1 librados: starting msgr at 2023-10-30T12:18:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.906+0000 7efe2be56640 1 librados: starting objecter 2023-10-30T12:18:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.906+0000 7efe2be56640 1 Processor -- start 2023-10-30T12:18:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.906+0000 7efe2be56640 1 -- start start 2023-10-30T12:18:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.906+0000 7efe2be56640 1 librados: setting wanted keys 2023-10-30T12:18:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.906+0000 7efe2be56640 1 librados: calling monclient init 2023-10-30T12:18:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.906+0000 7efe2be56640 10 monclient: init 2023-10-30T12:18:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe240b7c10 0x7efe241da180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 10 monclient(hunting): picked mon.noname-a con 0x7efe240b7c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe2413e730 0x7efe240ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 10 monclient(hunting): picked mon.noname-c con 0x7efe2413e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe24141940 0x7efe241383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 10 monclient(hunting): picked mon.noname-b con 0x7efe24141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe240b2fd0 con 0x7efe240b7c10 2023-10-30T12:18:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe241da6c0 con 0x7efe24141940 2023-10-30T12:18:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe24139650 con 0x7efe2413e730 2023-10-30T12:18:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.907+0000 7efe2be56640 10 monclient(hunting): authenticate will time out at 1578.112428s 2023-10-30T12:18:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe293f1640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe2413e730 0x7efe240ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe2a3f3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe24141940 0x7efe241383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe293f1640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe2413e730 0x7efe240ac300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54606/0 (socket says 172.21.15.148:54606) 2023-10-30T12:18:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe293f1640 1 -- 172.21.15.148:0/2037380485 learned_addr learned my addr 172.21.15.148:0/2037380485 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe293f1640 10 monclient(hunting): get_auth_request con 0x7efe2413e730 auth_method 0 2023-10-30T12:18:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe293f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe293f1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe293f1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe2a3f3640 10 monclient(hunting): get_auth_request con 0x7efe24141940 auth_method 0 2023-10-30T12:18:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe2a3f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe2a3f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe2a3f3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe293f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe293f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:05.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.908+0000 7efe293f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:05.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.909+0000 7efe2a3f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:05.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.909+0000 7efe2a3f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:05.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.909+0000 7efe2a3f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:05.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.912+0000 7efe293f1640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2023-10-30T12:18:05.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.912+0000 7efe293f1640 1 -- 172.21.15.148:0/2037380485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe24141940 msgr2=0x7efe241383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.912+0000 7efe293f1640 1 --2- 172.21.15.148:0/2037380485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe24141940 0x7efe241383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.912+0000 7efe293f1640 1 -- 172.21.15.148:0/2037380485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe240b7c10 msgr2=0x7efe241da180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:05.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.912+0000 7efe293f1640 1 --2- 172.21.15.148:0/2037380485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe240b7c10 0x7efe241da180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.912+0000 7efe293f1640 10 monclient: _finish_hunting 0 2023-10-30T12:18:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.912+0000 7efe293f1640 1 monclient: found mon.noname-c 2023-10-30T12:18:05.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.912+0000 7efe293f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:05.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.912+0000 7efe293f1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:18:05.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.912+0000 7efe293f1640 1 -- 172.21.15.148:0/2037380485 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe240ae0c0 con 0x7efe2413e730 2023-10-30T12:18:05.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.914+0000 7efe293f1640 1 --2- 172.21.15.148:0/2037380485 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe2413e730 0x7efe240ac300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efe1400b3e0 tx=0x7efe1400b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:05.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.914+0000 7efe1a7fc640 10 client.?.objecter ms_handle_connect 0x7efe2413e730 2023-10-30T12:18:05.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.914+0000 7efe1a7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:05.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 1 -- 172.21.15.148:0/2037380485 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14011830 con 0x7efe2413e730 2023-10-30T12:18:05.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:05.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:05.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 10 monclient: dump: 2023-10-30T12:18:05.951 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:05.952 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:05.952 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.952 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.953 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:05.953 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:05.954 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:05.954 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:05.954 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:05.955 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:05.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:05.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:05.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 1 -- 172.21.15.148:0/2037380485 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1400d040 con 0x7efe2413e730 2023-10-30T12:18:05.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe2be56640 5 monclient: authenticate success, global_id 4298 2023-10-30T12:18:05.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:05.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 1 -- 172.21.15.148:0/2037380485 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14011c00 con 0x7efe2413e730 2023-10-30T12:18:05.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:05.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:05.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.918+0000 7efe1a7fc640 10 monclient: dump: 2023-10-30T12:18:05.959 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:05.959 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:05.959 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.959 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:05.960 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:05.960 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:05.960 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:05.961 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:05.961 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:05.961 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:05.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.919+0000 7efe2be56640 10 monclient: _renew_subs 2023-10-30T12:18:05.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.919+0000 7efe2be56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:05.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.919+0000 7efe2be56640 1 -- 172.21.15.148:0/2037380485 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe240ac840 con 0x7efe2413e730 2023-10-30T12:18:05.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.919+0000 7efe1bfff640 4 set_mon_vals no callback set 2023-10-30T12:18:05.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.919+0000 7efe2be56640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:05.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.919+0000 7efe2be56640 10 monclient: _renew_subs 2023-10-30T12:18:05.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.919+0000 7efe2be56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:05.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.919+0000 7efe2be56640 1 -- 172.21.15.148:0/2037380485 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe241dace0 con 0x7efe2413e730 2023-10-30T12:18:05.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.919+0000 7efe2be56640 1 librados: init done 2023-10-30T12:18:05.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.920+0000 7efe2be56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:05.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.920+0000 7efe2be56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:05.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.920+0000 7efe2be56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:05.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:05.920+0000 7efe2be56640 1 -- 172.21.15.148:0/2037380485 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe240a97c0 con 0x7efe2413e730 2023-10-30T12:18:05.965 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:18:06.002 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T12:18:06.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.080+0000 7f3c0335a640 10 monclient: get_monmap_and_config 2023-10-30T12:18:06.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.080+0000 7f3c0335a640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.080+0000 7f3c0335a640 10 monclient: monmap: 2023-10-30T12:18:06.083 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.083 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.083 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:06.081734+0000 2023-10-30T12:18:06.083 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:06.081734+0000 2023-10-30T12:18:06.084 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.084 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.084 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:06.084 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:06.084 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:06.085 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:06.085 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:06.085 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.081+0000 7f3c0335a640 1 Processor -- start 2023-10-30T12:18:06.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.081+0000 7f3c0335a640 1 -- start start 2023-10-30T12:18:06.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.081+0000 7f3c0335a640 10 monclient: init 2023-10-30T12:18:06.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-10-30T12:18:06.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3bfc0b7c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:06.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3bfc141940 0x7f3bfc141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 10 monclient(hunting): picked mon.noname-c con 0x7f3bfc141940 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:06.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3bfc13e730 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:06.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0a9350 con 0x7f3bfc13e730 2023-10-30T12:18:06.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0aa6a0 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bfc0670a0 con 0x7f3bfc141940 2023-10-30T12:18:06.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c0335a640 10 monclient(hunting): authenticate will time out at 1578.287089s 2023-10-30T12:18:06.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.082+0000 7f3c008f5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3bfc141940 0x7f3bfc141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.083+0000 7f3c010f6640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3bfc0b7c10 0x7f3bfc0a8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:48310/0 (socket says 172.21.15.148:48310) 2023-10-30T12:18:06.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.083+0000 7f3c010f6640 1 -- 172.21.15.148:0/1971039910 learned_addr learned my addr 172.21.15.148:0/1971039910 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:18:06.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.083+0000 7f3c008f5640 10 monclient(hunting): get_auth_request con 0x7f3bfc141940 auth_method 0 2023-10-30T12:18:06.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.083+0000 7f3c008f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.083+0000 7f3c008f5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.083+0000 7f3c008f5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.084+0000 7f3c008f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.084+0000 7f3c008f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.084+0000 7f3c008f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.086+0000 7f3c008f5640 10 monclient(hunting): handle_auth_done global_id 4378 payload 274 2023-10-30T12:18:06.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.086+0000 7f3c008f5640 1 -- 172.21.15.148:0/1971039910 >> v1:172.21.15.150:6789/0 conn(0x7f3bfc0b7c10 legacy=0x7f3bfc0a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.086+0000 7f3c008f5640 1 -- 172.21.15.148:0/1971039910 >> v1:172.21.15.148:6789/0 conn(0x7f3bfc13e730 legacy=0x7f3bfc13eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:06.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.086+0000 7f3c008f5640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.086+0000 7f3c008f5640 1 monclient: found mon.noname-c 2023-10-30T12:18:06.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.086+0000 7f3c008f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.086+0000 7f3c008f5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.086+0000 7f3c008f5640 1 -- 172.21.15.148:0/1971039910 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc0aa910 con 0x7f3bfc141940 2023-10-30T12:18:06.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.091+0000 7f3c008f5640 1 --2- 172.21.15.148:0/1971039910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc141940 0x7f3bfc141d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0097a0 tx=0x7f3bec02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af377b790a3b5ce5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 1 -- 172.21.15.148:0/1971039910 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec0315a0 con 0x7f3bfc141940 2023-10-30T12:18:06.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:06.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 10 monclient: dump: 2023-10-30T12:18:06.097 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.097 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.098 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.098 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.098 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.098 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.099 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.099 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.099 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.099 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 1 -- 172.21.15.148:0/1971039910 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec031710 con 0x7f3bfc141940 2023-10-30T12:18:06.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 5 monclient: authenticate success, global_id 4378 2023-10-30T12:18:06.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:06.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:06.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 4 set_mon_vals no callback set 2023-10-30T12:18:06.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 1 -- 172.21.15.148:0/1971039910 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec031ab0 con 0x7f3bfc141940 2023-10-30T12:18:06.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:06.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3bf3fff640 10 monclient: dump: 2023-10-30T12:18:06.103 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.103 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.104 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.104 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.104 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.104 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.105 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.105 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.105 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.105 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 10 monclient: shutdown 2023-10-30T12:18:06.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:06.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 1 -- 172.21.15.148:0/1971039910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc141940 msgr2=0x7f3bfc141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 1 --2- 172.21.15.148:0/1971039910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc141940 0x7f3bfc141d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0097a0 tx=0x7f3bec02b840 comp rx=0 tx=0).stop 2023-10-30T12:18:06.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 1 -- 172.21.15.148:0/1971039910 shutdown_connections 2023-10-30T12:18:06.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 1 --2- 172.21.15.148:0/1971039910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc141940 0x7f3bfc141d90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.095+0000 7f3c0335a640 1 -- 172.21.15.148:0/1971039910 >> 172.21.15.148:0/1971039910 conn(0x7f3bfc06c660 msgr2=0x7f3bfc0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:06.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.096+0000 7f3c0335a640 1 -- 172.21.15.148:0/1971039910 shutdown_connections 2023-10-30T12:18:06.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.096+0000 7f3c0335a640 1 -- 172.21.15.148:0/1971039910 wait complete. 2023-10-30T12:18:06.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.096+0000 7f3c0335a640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.096+0000 7f3c0335a640 10 monclient: monmap: 2023-10-30T12:18:06.111 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.111 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:06.111 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:06.111 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:06.111 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.111 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:06.112 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:06.112 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:06.112 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:06.112 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.096+0000 7f3c0335a640 1 librados: starting msgr at 2023-10-30T12:18:06.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.096+0000 7f3c0335a640 1 librados: starting objecter 2023-10-30T12:18:06.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 1 Processor -- start 2023-10-30T12:18:06.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 1 -- start start 2023-10-30T12:18:06.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 1 librados: setting wanted keys 2023-10-30T12:18:06.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 1 librados: calling monclient init 2023-10-30T12:18:06.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient: init 2023-10-30T12:18:06.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:18:06.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc0b7c10 0x7f3bfc1da390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient(hunting): picked mon.noname-b con 0x7f3bfc0b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:06.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3bfc13e730 0x7f3bfc0abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient(hunting): picked mon.noname-a con 0x7f3bfc13e730 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:06.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3bfc141940 0x7f3bfc1da9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient(hunting): picked mon.noname-c con 0x7f3bfc141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:06.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc13ebd0 con 0x7f3bfc13e730 2023-10-30T12:18:06.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0ac490 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0b1020 con 0x7f3bfc141940 2023-10-30T12:18:06.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.097+0000 7f3c0335a640 10 monclient(hunting): authenticate will time out at 1578.302585s 2023-10-30T12:18:06.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.098+0000 7f3c010f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc0b7c10 0x7f3bfc1da390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.098+0000 7f3c010f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc0b7c10 0x7f3bfc1da390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48932/0 (socket says 172.21.15.148:48932) 2023-10-30T12:18:06.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.098+0000 7f3c010f6640 1 -- 172.21.15.148:0/2025422291 learned_addr learned my addr 172.21.15.148:0/2025422291 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:06.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.098+0000 7f3c010f6640 10 monclient(hunting): get_auth_request con 0x7f3bfc0b7c10 auth_method 0 2023-10-30T12:18:06.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.098+0000 7f3c010f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.098+0000 7f3c010f6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.098+0000 7f3c010f6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.099+0000 7f3c010f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.099+0000 7f3c010f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.099+0000 7f3c010f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.102+0000 7f3c010f6640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2023-10-30T12:18:06.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.102+0000 7f3c010f6640 1 -- 172.21.15.148:0/2025422291 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3bfc141940 msgr2=0x7f3bfc1da9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.102+0000 7f3c010f6640 1 --2- 172.21.15.148:0/2025422291 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3bfc141940 0x7f3bfc1da9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.102+0000 7f3c010f6640 1 -- 172.21.15.148:0/2025422291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3bfc13e730 msgr2=0x7f3bfc0abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.102+0000 7f3c010f6640 1 --2- 172.21.15.148:0/2025422291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3bfc13e730 0x7f3bfc0abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.102+0000 7f3c010f6640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.102+0000 7f3c010f6640 1 monclient: found mon.noname-b 2023-10-30T12:18:06.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.102+0000 7f3c010f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.102+0000 7f3c010f6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.102+0000 7f3c010f6640 1 -- 172.21.15.148:0/2025422291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc1daf10 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.104+0000 7f3c010f6640 1 --2- 172.21.15.148:0/2025422291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc0b7c10 0x7f3bfc1da390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800c4a0 tx=0x7f3bf800c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.104+0000 7f3bf1ffb640 10 client.?.objecter ms_handle_connect 0x7f3bfc0b7c10 2023-10-30T12:18:06.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.104+0000 7f3bf1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:06.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 1 -- 172.21.15.148:0/2025422291 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8005ce0 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:06.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 10 monclient: dump: 2023-10-30T12:18:06.125 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.125 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.125 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.126 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.126 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.126 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.126 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.127 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.127 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.127 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 1 -- 172.21.15.148:0/2025422291 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf800e040 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3c0335a640 5 monclient: authenticate success, global_id 4384 2023-10-30T12:18:06.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3c0335a640 10 monclient: _renew_subs 2023-10-30T12:18:06.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3c0335a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3c0335a640 1 -- 172.21.15.148:0/2025422291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc0add80 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 1 -- 172.21.15.148:0/2025422291 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8005e90 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:06.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf1ffb640 10 monclient: dump: 2023-10-30T12:18:06.131 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.131 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.131 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.131 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.131 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.132 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.132 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.132 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.132 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.133 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3bf37fe640 4 set_mon_vals no callback set 2023-10-30T12:18:06.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3c0335a640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:06.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3c0335a640 10 monclient: _renew_subs 2023-10-30T12:18:06.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3c0335a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3c0335a640 1 -- 172.21.15.148:0/2025422291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc1392e0 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.108+0000 7f3c0335a640 1 librados: init done 2023-10-30T12:18:06.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.109+0000 7f3c0335a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.109+0000 7f3c0335a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.109+0000 7f3c0335a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.109+0000 7f3c0335a640 1 -- 172.21.15.148:0/2025422291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc142290 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.118+0000 7f7a56661640 10 monclient: get_monmap_and_config 2023-10-30T12:18:06.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.119+0000 7f7a56661640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.119+0000 7f7a56661640 10 monclient: monmap: 2023-10-30T12:18:06.137 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.137 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.138 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:06.120060+0000 2023-10-30T12:18:06.138 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:06.120060+0000 2023-10-30T12:18:06.138 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.139 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.139 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:06.139 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:06.139 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:06.140 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:06.140 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:06.140 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.119+0000 7f7a56661640 1 Processor -- start 2023-10-30T12:18:06.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.119+0000 7f7a56661640 1 -- start start 2023-10-30T12:18:06.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.119+0000 7f7a56661640 10 monclient: init 2023-10-30T12:18:06.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-10-30T12:18:06.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a500b7c10 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:06.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7a50141940 0x7f7a50141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 10 monclient(hunting): picked mon.noname-a con 0x7f7a50141940 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:06.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a5013e730 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:06.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a500a9350 con 0x7f7a500b7c10 2023-10-30T12:18:06.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a500aa6a0 con 0x7f7a5013e730 2023-10-30T12:18:06.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a500670a0 con 0x7f7a50141940 2023-10-30T12:18:06.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.120+0000 7f7a56661640 10 monclient(hunting): authenticate will time out at 1578.325289s 2023-10-30T12:18:06.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.126+0000 7f3bf1ffb640 1 -- 172.21.15.148:0/2025422291 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f3bf8007270 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.127+0000 7f3bf1ffb640 1 --2- 172.21.15.148:0/2025422291 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3be006c7c0 0x7f3be006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.127+0000 7f3c008f5640 1 --2- 172.21.15.148:0/2025422291 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3be006c7c0 0x7f3be006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.127+0000 7f3c008f5640 10 monclient: get_auth_request con 0x7f3be006c7c0 auth_method 0 2023-10-30T12:18:06.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.127+0000 7f3c008f5640 1 --2- 172.21.15.148:0/2025422291 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3be006c7c0 0x7f3be006ec80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3bec036000 tx=0x7f3bec030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.127+0000 7f3bf1ffb640 10 client.4384.objecter ms_handle_connect 0x7f3be006c7c0 2023-10-30T12:18:06.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.129+0000 7f3bf1ffb640 1 -- 172.21.15.148:0/2025422291 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f3bf8057740 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.129+0000 7f3bf1ffb640 10 client.4384.objecter ms_dispatch 0x7f3bfc000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:06.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.129+0000 7f3bf1ffb640 3 client.4384.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:06.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.129+0000 7f3bf1ffb640 3 client.4384.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:06.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.129+0000 7f3bf1ffb640 20 client.4384.objecter dump_active .. 0 homeless 2023-10-30T12:18:06.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.130+0000 7f7a54bfe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7a5013e730 0x7f7a5013eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:48316/0 (socket says 172.21.15.148:48316) 2023-10-30T12:18:06.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.130+0000 7f7a54bfe640 1 -- 172.21.15.148:0/1228658394 learned_addr learned my addr 172.21.15.148:0/1228658394 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:18:06.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.137+0000 7fccdcff9640 1 -- 172.21.15.148:0/3608585370 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fccd4057f10 con 0x7fcce013e730 2023-10-30T12:18:06.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.137+0000 7fccdcff9640 10 client.4335.objecter ms_dispatch 0x7fcce0000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:06.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.137+0000 7fccdcff9640 3 client.4335.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:06.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.137+0000 7fccdcff9640 3 client.4335.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:06.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.137+0000 7fccdcff9640 20 client.4335.objecter dump_active .. 0 homeless 2023-10-30T12:18:06.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.137+0000 7fccdcff9640 1 -- 172.21.15.148:0/3608585370 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fccd4016070 con 0x7fcce013e730 2023-10-30T12:18:06.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.137+0000 7fccdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.137+0000 7fccdcff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:06.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.144+0000 7f7a4f7fe640 1 --2- 172.21.15.148:0/1228658394 >> v2:172.21.15.148:3300/0 conn(0x7f7a50141940 0x7f7a50141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.144+0000 7f7a4f7fe640 10 monclient(hunting): get_auth_request con 0x7f7a50141940 auth_method 0 2023-10-30T12:18:06.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.144+0000 7f7a4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.144+0000 7f7a4f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.144+0000 7f7a4f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.145+0000 7f7a4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.145+0000 7f7a4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.145+0000 7f7a4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.148+0000 7f7a4f7fe640 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2023-10-30T12:18:06.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.148+0000 7f7a4f7fe640 1 -- 172.21.15.148:0/1228658394 >> v1:172.21.15.150:6789/0 conn(0x7f7a5013e730 legacy=0x7f7a5013eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:06.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.148+0000 7f7a4f7fe640 1 -- 172.21.15.148:0/1228658394 >> v1:172.21.15.148:6789/0 conn(0x7f7a500b7c10 legacy=0x7f7a500a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.148+0000 7f7a4f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.148+0000 7f7a4f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:18:06.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.148+0000 7f7a4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.148+0000 7f7a4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.148+0000 7f7a4f7fe640 1 -- 172.21.15.148:0/1228658394 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a500aa910 con 0x7f7a50141940 2023-10-30T12:18:06.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.153+0000 7f7a4f7fe640 1 --2- 172.21.15.148:0/1228658394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a50141940 0x7f7a50141d90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a440097a0 tx=0x7f7a4402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fec7022b7bab584 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 1 -- 172.21.15.148:0/1228658394 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a440315a0 con 0x7f7a50141940 2023-10-30T12:18:06.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:06.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 10 monclient: dump: 2023-10-30T12:18:06.163 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.163 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.163 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.164 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.164 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.164 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.165 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.165 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.165 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.165 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 1 -- 172.21.15.148:0/1228658394 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a44031710 con 0x7f7a50141940 2023-10-30T12:18:06.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a56661640 5 monclient: authenticate success, global_id 4338 2023-10-30T12:18:06.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a56661640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:06.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a56661640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:06.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a56661640 4 set_mon_vals no callback set 2023-10-30T12:18:06.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 1 -- 172.21.15.148:0/1228658394 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a44031ab0 con 0x7f7a50141940 2023-10-30T12:18:06.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:06.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.160+0000 7f7a4effd640 10 monclient: dump: 2023-10-30T12:18:06.169 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.169 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.169 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.169 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.170 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.170 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.170 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.170 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.171 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.171 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.161+0000 7f7a56661640 10 monclient: shutdown 2023-10-30T12:18:06.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.161+0000 7f7a56661640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:06.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.161+0000 7f7a56661640 1 -- 172.21.15.148:0/1228658394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a50141940 msgr2=0x7f7a50141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.161+0000 7f7a56661640 1 --2- 172.21.15.148:0/1228658394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a50141940 0x7f7a50141d90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a440097a0 tx=0x7f7a4402b840 comp rx=0 tx=0).stop 2023-10-30T12:18:06.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.161+0000 7f7a56661640 1 -- 172.21.15.148:0/1228658394 shutdown_connections 2023-10-30T12:18:06.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.161+0000 7f7a56661640 1 --2- 172.21.15.148:0/1228658394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a50141940 0x7f7a50141d90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.161+0000 7f7a56661640 1 -- 172.21.15.148:0/1228658394 >> 172.21.15.148:0/1228658394 conn(0x7f7a5006c660 msgr2=0x7f7a500b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:06.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.161+0000 7f7a56661640 1 -- 172.21.15.148:0/1228658394 shutdown_connections 2023-10-30T12:18:06.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.161+0000 7f7a56661640 1 -- 172.21.15.148:0/1228658394 wait complete. 2023-10-30T12:18:06.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.162+0000 7f7a56661640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.162+0000 7f7a56661640 10 monclient: monmap: 2023-10-30T12:18:06.176 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.177 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:06.177 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:06.177 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:06.177 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.178 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:06.178 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:06.178 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:06.178 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:06.178 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.162+0000 7f7a56661640 1 librados: starting msgr at 2023-10-30T12:18:06.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.162+0000 7f7a56661640 1 librados: starting objecter 2023-10-30T12:18:06.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.162+0000 7f7a56661640 1 Processor -- start 2023-10-30T12:18:06.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.162+0000 7f7a56661640 1 -- start start 2023-10-30T12:18:06.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.162+0000 7f7a56661640 1 librados: setting wanted keys 2023-10-30T12:18:06.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.162+0000 7f7a56661640 1 librados: calling monclient init 2023-10-30T12:18:06.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.162+0000 7f7a56661640 10 monclient: init 2023-10-30T12:18:06.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:18:06.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a500b7c10 0x7f7a501da300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 10 monclient(hunting): picked mon.noname-a con 0x7f7a500b7c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:06.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a5013e730 0x7f7a500ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 10 monclient(hunting): picked mon.noname-b con 0x7f7a5013e730 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:06.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7a50141940 0x7f7a500ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 10 monclient(hunting): picked mon.noname-c con 0x7f7a50141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:06.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a50073a60 con 0x7f7a500b7c10 2023-10-30T12:18:06.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a501dafd0 con 0x7f7a5013e730 2023-10-30T12:18:06.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a501dab40 con 0x7f7a50141940 2023-10-30T12:18:06.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.163+0000 7f7a56661640 10 monclient(hunting): authenticate will time out at 1578.368230s 2023-10-30T12:18:06.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.170+0000 7f7a4ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a500b7c10 0x7f7a501da300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.171+0000 7f7a4ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a500b7c10 0x7f7a501da300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60076/0 (socket says 172.21.15.148:60076) 2023-10-30T12:18:06.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.171+0000 7f7a4ffff640 1 -- 172.21.15.148:0/236113376 learned_addr learned my addr 172.21.15.148:0/236113376 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:06.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.171+0000 7f7a4ffff640 10 monclient(hunting): get_auth_request con 0x7f7a500b7c10 auth_method 0 2023-10-30T12:18:06.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.171+0000 7f7a4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.171+0000 7f7a4ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.171+0000 7f7a4ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.174+0000 7f7a4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.174+0000 7f7a4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.174+0000 7f7a4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.177+0000 7f7a4ffff640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2023-10-30T12:18:06.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.177+0000 7f7a4ffff640 1 -- 172.21.15.148:0/236113376 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7a50141940 msgr2=0x7f7a500ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.177+0000 7f7a4ffff640 1 --2- 172.21.15.148:0/236113376 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7a50141940 0x7f7a500ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.177+0000 7f7a4ffff640 1 -- 172.21.15.148:0/236113376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a5013e730 msgr2=0x7f7a500ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.177+0000 7f7a4ffff640 1 --2- 172.21.15.148:0/236113376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a5013e730 0x7f7a500ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.177+0000 7f7a4ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.177+0000 7f7a4ffff640 1 monclient: found mon.noname-a 2023-10-30T12:18:06.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.177+0000 7f7a4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.177+0000 7f7a4ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.177+0000 7f7a4ffff640 1 -- 172.21.15.148:0/236113376 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a501da870 con 0x7f7a500b7c10 2023-10-30T12:18:06.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.179+0000 7f7a4ffff640 1 --2- 172.21.15.148:0/236113376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a500b7c10 0x7f7a501da300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000c450 tx=0x7f7a4000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.179+0000 7f7a4cff9640 10 client.?.objecter ms_handle_connect 0x7f7a500b7c10 2023-10-30T12:18:06.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.179+0000 7f7a4cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:06.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 1 -- 172.21.15.148:0/236113376 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a400052d0 con 0x7f7a500b7c10 2023-10-30T12:18:06.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:06.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 10 monclient: dump: 2023-10-30T12:18:06.191 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.191 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.191 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.191 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.191 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.192 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.192 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.192 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.192 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.192 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 1 -- 172.21.15.148:0/236113376 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4000e040 con 0x7f7a500b7c10 2023-10-30T12:18:06.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a56661640 5 monclient: authenticate success, global_id 4344 2023-10-30T12:18:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a56661640 10 monclient: _renew_subs 2023-10-30T12:18:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 1 -- 172.21.15.148:0/236113376 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a400035f0 con 0x7f7a500b7c10 2023-10-30T12:18:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a56661640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a56661640 1 -- 172.21.15.148:0/236113376 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a500adf70 con 0x7f7a500b7c10 2023-10-30T12:18:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:06.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4cff9640 10 monclient: dump: 2023-10-30T12:18:06.195 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.195 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.196 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.196 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.196 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.197 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.197 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.197 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.197 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.197 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a4e7fc640 4 set_mon_vals no callback set 2023-10-30T12:18:06.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a56661640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:06.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a56661640 10 monclient: _renew_subs 2023-10-30T12:18:06.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a56661640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a56661640 1 -- 172.21.15.148:0/236113376 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a50138240 con 0x7f7a500b7c10 2023-10-30T12:18:06.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.183+0000 7f7a56661640 1 librados: init done 2023-10-30T12:18:06.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.184+0000 7f7a56661640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.184+0000 7f7a56661640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.184+0000 7f7a56661640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.184+0000 7f7a56661640 1 -- 172.21.15.148:0/236113376 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a500aa350 con 0x7f7a500b7c10 2023-10-30T12:18:06.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.202+0000 7efe1a7fc640 1 -- 172.21.15.148:0/2037380485 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7efe1401a400 con 0x7efe2413e730 2023-10-30T12:18:06.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.202+0000 7efe1a7fc640 1 --2- 172.21.15.148:0/2037380485 >> v2:172.21.15.148:6800/3205090179 conn(0x7efe0806c7f0 0x7efe0806ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.203+0000 7efe29bf2640 1 --2- 172.21.15.148:0/2037380485 >> v2:172.21.15.148:6800/3205090179 conn(0x7efe0806c7f0 0x7efe0806ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.203+0000 7efe29bf2640 10 monclient: get_auth_request con 0x7efe0806c7f0 auth_method 0 2023-10-30T12:18:06.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.203+0000 7efe1a7fc640 10 client.4298.objecter ms_handle_connect 0x7efe0806c7f0 2023-10-30T12:18:06.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.203+0000 7efe29bf2640 1 --2- 172.21.15.148:0/2037380485 >> v2:172.21.15.148:6800/3205090179 conn(0x7efe0806c7f0 0x7efe0806ecb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efe20036000 tx=0x7efe20030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.205+0000 7f7a4cff9640 1 -- 172.21.15.148:0/236113376 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f7a40003760 con 0x7f7a500b7c10 2023-10-30T12:18:06.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.206+0000 7f7a4cff9640 1 --2- 172.21.15.148:0/236113376 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7a3c06c7c0 0x7f7a3c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.206+0000 7f7a4f7fe640 1 --2- 172.21.15.148:0/236113376 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7a3c06c7c0 0x7f7a3c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.206+0000 7f7a4f7fe640 10 monclient: get_auth_request con 0x7f7a3c06c7c0 auth_method 0 2023-10-30T12:18:06.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.207+0000 7f7a4cff9640 10 client.4344.objecter ms_handle_connect 0x7f7a3c06c7c0 2023-10-30T12:18:06.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.207+0000 7f7a4f7fe640 1 --2- 172.21.15.148:0/236113376 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7a3c06c7c0 0x7f7a3c06ec80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7a44036000 tx=0x7f7a44030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.211+0000 7f7a4cff9640 1 -- 172.21.15.148:0/236113376 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f7a40057e50 con 0x7f7a500b7c10 2023-10-30T12:18:06.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.211+0000 7f7a4cff9640 10 client.4344.objecter ms_dispatch 0x7f7a50000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:06.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.211+0000 7f7a4cff9640 3 client.4344.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:06.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.211+0000 7f7a4cff9640 3 client.4344.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:06.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.211+0000 7f7a4cff9640 20 client.4344.objecter dump_active .. 0 homeless 2023-10-30T12:18:06.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.226+0000 7efe1a7fc640 1 -- 172.21.15.148:0/2037380485 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7efe14058920 con 0x7efe2413e730 2023-10-30T12:18:06.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.226+0000 7efe1a7fc640 10 client.4298.objecter ms_dispatch 0x7efe24000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:06.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.226+0000 7efe1a7fc640 3 client.4298.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:06.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.226+0000 7efe1a7fc640 3 client.4298.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:06.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.226+0000 7efe1a7fc640 20 client.4298.objecter dump_active .. 0 homeless 2023-10-30T12:18:06.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.226+0000 7efe1a7fc640 1 -- 172.21.15.148:0/2037380485 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efe14058220 con 0x7efe2413e730 2023-10-30T12:18:06.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.226+0000 7efe1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.226+0000 7efe1a7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.350+0000 7fcce63f0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:18:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.350+0000 7fcce63f0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:18:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.350+0000 7fcce63f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.350+0000 7fcce63f0640 1 -- 172.21.15.148:0/3608585370 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fcce00af3c0 con 0x7fcce013e730 2023-10-30T12:18:06.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.373+0000 7f3bf1ffb640 1 -- 172.21.15.148:0/2025422291 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3bf8086e50 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.373+0000 7f3bf1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.373+0000 7f3bf1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:06.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.403+0000 7efe2be56640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:18:06.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.403+0000 7efe2be56640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:18:06.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.403+0000 7efe2be56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:06.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.403+0000 7efe2be56640 1 -- 172.21.15.148:0/2037380485 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7efe240ac840 con 0x7efe2413e730 2023-10-30T12:18:06.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.408+0000 7efe1a7fc640 1 -- 172.21.15.148:0/2037380485 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efe1405c370 con 0x7efe2413e730 2023-10-30T12:18:06.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.408+0000 7efe1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:18:06.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.408+0000 7efe1a7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:06.411 INFO:teuthology.orchestra.run.smithi148.stdout:64424509445 2023-10-30T12:18:06.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.410+0000 7efe2be56640 10 librados: watch_flush enter 2023-10-30T12:18:06.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.410+0000 7efe2be56640 10 librados: watch_flush exit 2023-10-30T12:18:06.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.410+0000 7efe2be56640 20 client.4298.objecter shutdown clearing up homeless session... 2023-10-30T12:18:06.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.410+0000 7efe2be56640 10 client.4298.objecter successfully canceled tick 2023-10-30T12:18:06.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.410+0000 7efe2be56640 1 -- 172.21.15.148:0/2037380485 >> v2:172.21.15.148:6800/3205090179 conn(0x7efe0806c7f0 msgr2=0x7efe0806ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.410+0000 7efe2be56640 1 --2- 172.21.15.148:0/2037380485 >> v2:172.21.15.148:6800/3205090179 conn(0x7efe0806c7f0 0x7efe0806ecb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efe20036000 tx=0x7efe20030000 comp rx=0 tx=0).stop 2023-10-30T12:18:06.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.410+0000 7efe2be56640 10 monclient: shutdown 2023-10-30T12:18:06.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.410+0000 7efe2be56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:06.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.410+0000 7efe2be56640 1 -- 172.21.15.148:0/2037380485 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe2413e730 msgr2=0x7efe240ac300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.410+0000 7efe2be56640 1 --2- 172.21.15.148:0/2037380485 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe2413e730 0x7efe240ac300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efe1400b3e0 tx=0x7efe1400b8a0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.411+0000 7efe2be56640 1 -- 172.21.15.148:0/2037380485 shutdown_connections 2023-10-30T12:18:06.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.411+0000 7efe2be56640 1 --2- 172.21.15.148:0/2037380485 >> v2:172.21.15.148:6800/3205090179 conn(0x7efe0806c7f0 0x7efe0806ecb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.411+0000 7efe2be56640 1 --2- 172.21.15.148:0/2037380485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe24141940 0x7efe241383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.411+0000 7efe2be56640 1 --2- 172.21.15.148:0/2037380485 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efe2413e730 0x7efe240ac300 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.411+0000 7efe2be56640 1 --2- 172.21.15.148:0/2037380485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe240b7c10 0x7efe241da180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.411+0000 7efe2be56640 1 -- 172.21.15.148:0/2037380485 >> 172.21.15.148:0/2037380485 conn(0x7efe2406c660 msgr2=0x7efe240b4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:06.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.412+0000 7efe2be56640 1 -- 172.21.15.148:0/2037380485 shutdown_connections 2023-10-30T12:18:06.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.412+0000 7efe2be56640 1 -- 172.21.15.148:0/2037380485 wait complete. 2023-10-30T12:18:06.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.412+0000 7efe2be56640 1 librados: shutdown 2023-10-30T12:18:06.425 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.4 2023-10-30T12:18:06.427 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T12:18:06.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.465+0000 7f7a4cff9640 1 -- 172.21.15.148:0/236113376 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a40057750 con 0x7f7a500b7c10 2023-10-30T12:18:06.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.466+0000 7f7a4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.466+0000 7f7a4cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:06.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.499+0000 7fccdcff9640 1 -- 172.21.15.148:0/3608585370 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fccd405bfb0 con 0x7fcce013e730 2023-10-30T12:18:06.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.499+0000 7fccdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:18:06.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.499+0000 7fccdcff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:06.501 INFO:teuthology.orchestra.run.smithi148.stdout:64424509445 2023-10-30T12:18:06.502 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T12:18:06.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 10 librados: watch_flush enter 2023-10-30T12:18:06.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 10 librados: watch_flush exit 2023-10-30T12:18:06.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 20 client.4335.objecter shutdown clearing up homeless session... 2023-10-30T12:18:06.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 10 client.4335.objecter successfully canceled tick 2023-10-30T12:18:06.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 1 -- 172.21.15.148:0/3608585370 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcccc06c7c0 msgr2=0x7fcccc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 1 --2- 172.21.15.148:0/3608585370 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcccc06c7c0 0x7fcccc06ec80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fccd0006fd0 tx=0x7fccd0008040 comp rx=0 tx=0).stop 2023-10-30T12:18:06.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 10 monclient: shutdown 2023-10-30T12:18:06.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:06.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 1 -- 172.21.15.148:0/3608585370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcce013e730 msgr2=0x7fcce01ce020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 1 --2- 172.21.15.148:0/3608585370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcce013e730 0x7fcce01ce020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccd400b420 tx=0x7fccd400b8e0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 1 -- 172.21.15.148:0/3608585370 shutdown_connections 2023-10-30T12:18:06.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 1 --2- 172.21.15.148:0/3608585370 >> v2:172.21.15.148:6800/3205090179 conn(0x7fcccc06c7c0 0x7fcccc06ec80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 1 --2- 172.21.15.148:0/3608585370 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fcce0141940 0x7fcce01ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 1 --2- 172.21.15.148:0/3608585370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcce013e730 0x7fcce01ce020 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 1 --2- 172.21.15.148:0/3608585370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcce00b7c10 0x7fcce00aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.501+0000 7fcce63f0640 1 -- 172.21.15.148:0/3608585370 >> 172.21.15.148:0/3608585370 conn(0x7fcce006c660 msgr2=0x7fcce00b3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:06.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.502+0000 7fcce63f0640 1 -- 172.21.15.148:0/3608585370 shutdown_connections 2023-10-30T12:18:06.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.502+0000 7fcce63f0640 1 -- 172.21.15.148:0/3608585370 wait complete. 2023-10-30T12:18:06.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.502+0000 7fcce63f0640 1 librados: shutdown 2023-10-30T12:18:06.512 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T12:18:06.517 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T12:18:06.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.5 2023-10-30T12:18:06.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.544+0000 7fc9ecdbb640 10 monclient: get_monmap_and_config 2023-10-30T12:18:06.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.544+0000 7fc9ecdbb640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.544+0000 7fc9ecdbb640 10 monclient: monmap: 2023-10-30T12:18:06.546 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.547 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.547 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:06.545560+0000 2023-10-30T12:18:06.547 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:06.545560+0000 2023-10-30T12:18:06.547 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.547 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.548 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:06.548 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:06.548 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:06.548 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:06.548 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:06.548 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.545+0000 7fc9ecdbb640 1 Processor -- start 2023-10-30T12:18:06.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.545+0000 7fc9ecdbb640 1 -- start start 2023-10-30T12:18:06.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.545+0000 7fc9ecdbb640 10 monclient: init 2023-10-30T12:18:06.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T12:18:06.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc9e80b7c10 0x7fc9e80a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 10 monclient(hunting): picked mon.noname-c con 0x7fc9e80b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:06.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc9e8141940 0x7fc9e8141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 10 monclient(hunting): picked mon.noname-a con 0x7fc9e8141940 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:06.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b1c10 0x7fc9e80b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e80b1c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:06.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e80670a0 con 0x7fc9e8141940 2023-10-30T12:18:06.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e80b7ff0 con 0x7fc9e80b7c10 2023-10-30T12:18:06.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e80a9350 con 0x7fc9e80b1c10 2023-10-30T12:18:06.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.546+0000 7fc9ecdbb640 10 monclient(hunting): authenticate will time out at 1578.751634s 2023-10-30T12:18:06.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.547+0000 7fc9e6d9d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b1c10 0x7fc9e80b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.548+0000 7fc9e6d9d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b1c10 0x7fc9e80b2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54614/0 (socket says 172.21.15.148:54614) 2023-10-30T12:18:06.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.548+0000 7fc9e6d9d640 1 -- 172.21.15.148:0/2617958551 learned_addr learned my addr 172.21.15.148:0/2617958551 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:06.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.548+0000 7fc9e6d9d640 10 monclient(hunting): get_auth_request con 0x7fc9e80b1c10 auth_method 0 2023-10-30T12:18:06.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.548+0000 7fc9e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.548+0000 7fc9e6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.548+0000 7fc9e6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.550+0000 7fc9e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.550+0000 7fc9e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.550+0000 7fc9e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.556+0000 7fc9e659c640 1 --2- 172.21.15.148:0/2617958551 >> v2:172.21.15.150:3300/0 conn(0x7fc9e80b7c10 0x7fc9e80a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.556+0000 7fc9e659c640 10 monclient(hunting): get_auth_request con 0x7fc9e80b7c10 auth_method 0 2023-10-30T12:18:06.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.556+0000 7fc9e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.556+0000 7fc9e659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.556+0000 7fc9e659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.558+0000 7fc9e6d9d640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2023-10-30T12:18:06.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.558+0000 7fc9e6d9d640 1 -- 172.21.15.148:0/2617958551 >> v2:172.21.15.150:3300/0 conn(0x7fc9e80b7c10 msgr2=0x7fc9e80a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:06.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.558+0000 7fc9e6d9d640 1 --2- 172.21.15.148:0/2617958551 >> v2:172.21.15.150:3300/0 conn(0x7fc9e80b7c10 0x7fc9e80a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.558+0000 7fc9e6d9d640 1 -- 172.21.15.148:0/2617958551 >> v2:172.21.15.148:3300/0 conn(0x7fc9e8141940 msgr2=0x7fc9e8141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:06.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.558+0000 7fc9e6d9d640 1 --2- 172.21.15.148:0/2617958551 >> v2:172.21.15.148:3300/0 conn(0x7fc9e8141940 0x7fc9e8141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.558+0000 7fc9e6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.558+0000 7fc9e6d9d640 1 monclient: found mon.noname-b 2023-10-30T12:18:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.558+0000 7fc9e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.558+0000 7fc9e6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:06.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.558+0000 7fc9e6d9d640 1 -- 172.21.15.148:0/2617958551 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e80aa6a0 con 0x7fc9e80b1c10 2023-10-30T12:18:06.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.562+0000 7fc9e6d9d640 1 --2- 172.21.15.148:0/2617958551 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b1c10 0x7fc9e80b2060 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80b4d80 tx=0x7fc9e002be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=429330861061e39f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.586+0000 7f3c0335a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:18:06.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.586+0000 7f3c0335a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:18:06.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.586+0000 7f3c0335a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.586+0000 7f3c0335a640 1 -- 172.21.15.148:0/2025422291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3bfc0add80 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.590+0000 7f3bf1ffb640 1 -- 172.21.15.148:0/2025422291 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3bf8007520 con 0x7f3bfc0b7c10 2023-10-30T12:18:06.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.590+0000 7f3bf1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:18:06.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.590+0000 7f3bf1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:06.592 INFO:teuthology.orchestra.run.smithi148.stdout:64424509445 2023-10-30T12:18:06.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 10 librados: watch_flush enter 2023-10-30T12:18:06.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 10 librados: watch_flush exit 2023-10-30T12:18:06.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 20 client.4384.objecter shutdown clearing up homeless session... 2023-10-30T12:18:06.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 10 client.4384.objecter successfully canceled tick 2023-10-30T12:18:06.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 -- 172.21.15.148:0/2025422291 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3be006c7c0 msgr2=0x7f3be006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 --2- 172.21.15.148:0/2025422291 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3be006c7c0 0x7f3be006ec80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3bec036000 tx=0x7f3bec030000 comp rx=0 tx=0).stop 2023-10-30T12:18:06.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 10 monclient: shutdown 2023-10-30T12:18:06.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:06.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 -- 172.21.15.148:0/2025422291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc0b7c10 msgr2=0x7f3bfc1da390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 --2- 172.21.15.148:0/2025422291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc0b7c10 0x7f3bfc1da390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800c4a0 tx=0x7f3bf800c960 comp rx=0 tx=0).stop 2023-10-30T12:18:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 -- 172.21.15.148:0/2025422291 shutdown_connections 2023-10-30T12:18:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 --2- 172.21.15.148:0/2025422291 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3be006c7c0 0x7f3be006ec80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 --2- 172.21.15.148:0/2025422291 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3bfc141940 0x7f3bfc1da9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 --2- 172.21.15.148:0/2025422291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3bfc13e730 0x7f3bfc0abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 --2- 172.21.15.148:0/2025422291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bfc0b7c10 0x7f3bfc1da390 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 -- 172.21.15.148:0/2025422291 >> 172.21.15.148:0/2025422291 conn(0x7f3bfc06c660 msgr2=0x7f3bfc13bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:06.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.594+0000 7f3c0335a640 1 -- 172.21.15.148:0/2025422291 shutdown_connections 2023-10-30T12:18:06.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.595+0000 7f3c0335a640 1 -- 172.21.15.148:0/2025422291 wait complete. 2023-10-30T12:18:06.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.595+0000 7f3c0335a640 1 librados: shutdown 2023-10-30T12:18:06.616 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.2 2023-10-30T12:18:06.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.656+0000 7f1304589640 10 monclient: get_monmap_and_config 2023-10-30T12:18:06.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.656+0000 7f1304589640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.656+0000 7f1304589640 10 monclient: monmap: 2023-10-30T12:18:06.658 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.658 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.659 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:06.657373+0000 2023-10-30T12:18:06.659 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:06.657373+0000 2023-10-30T12:18:06.659 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.659 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.659 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:06.660 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:06.660 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:06.660 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:06.660 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:06.660 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.657+0000 7f1304589640 1 Processor -- start 2023-10-30T12:18:06.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.657+0000 7f1304589640 1 -- start start 2023-10-30T12:18:06.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.657+0000 7f1304589640 10 monclient: init 2023-10-30T12:18:06.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-10-30T12:18:06.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f12fc0b7c10 0x7f12fc0a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 10 monclient(hunting): picked mon.noname-a con 0x7f12fc0b7c10 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:06.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f12fc141940 0x7f12fc141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 10 monclient(hunting): picked mon.noname-b con 0x7f12fc141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:06.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12fc0b1c10 0x7f12fc0b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 10 monclient(hunting): picked mon.noname-c con 0x7f12fc0b1c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:06.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12fc0670a0 con 0x7f12fc0b7c10 2023-10-30T12:18:06.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12fc0b7ff0 con 0x7f12fc0b1c10 2023-10-30T12:18:06.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc0a9350 con 0x7f12fc141940 2023-10-30T12:18:06.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.658+0000 7f1304589640 10 monclient(hunting): authenticate will time out at 1578.863288s 2023-10-30T12:18:06.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302b26640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12fc0b1c10 0x7f12fc0b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302325640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f12fc0b7c10 0x7f12fc0a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302325640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f12fc0b7c10 0x7f12fc0a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60106/0 (socket says 172.21.15.148:60106) 2023-10-30T12:18:06.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302325640 1 -- 172.21.15.148:0/4181650145 learned_addr learned my addr 172.21.15.148:0/4181650145 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:06.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302325640 10 monclient(hunting): get_auth_request con 0x7f12fc0b7c10 auth_method 0 2023-10-30T12:18:06.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302325640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302325640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302325640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302b26640 10 monclient(hunting): get_auth_request con 0x7f12fc0b1c10 auth_method 0 2023-10-30T12:18:06.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302b26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302b26640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.659+0000 7f1302b26640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.660+0000 7f1302325640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.660+0000 7f1302325640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.660+0000 7f1302325640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.660+0000 7f1302b26640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.660+0000 7f1302b26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.660+0000 7f1302b26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.660+0000 7f1301b24640 1 --2- 172.21.15.148:0/4181650145 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f12fc141940 0x7f12fc141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.661+0000 7f1301b24640 10 monclient(hunting): get_auth_request con 0x7f12fc141940 auth_method 0 2023-10-30T12:18:06.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.661+0000 7f1301b24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.661+0000 7f1301b24640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.661+0000 7f1301b24640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.662+0000 7f1302b26640 10 monclient(hunting): handle_auth_done global_id 4390 payload 274 2023-10-30T12:18:06.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.662+0000 7f1302b26640 1 -- 172.21.15.148:0/4181650145 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f12fc141940 msgr2=0x7f12fc141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:06.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.662+0000 7f1302b26640 1 --2- 172.21.15.148:0/4181650145 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f12fc141940 0x7f12fc141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.662+0000 7f1302b26640 1 -- 172.21.15.148:0/4181650145 >> v2:172.21.15.148:3300/0 conn(0x7f12fc0b7c10 msgr2=0x7f12fc0a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:06.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.662+0000 7f1302b26640 1 --2- 172.21.15.148:0/4181650145 >> v2:172.21.15.148:3300/0 conn(0x7f12fc0b7c10 0x7f12fc0a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.662+0000 7f1302b26640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.662+0000 7f1302b26640 1 monclient: found mon.noname-c 2023-10-30T12:18:06.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.662+0000 7f1302b26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.662+0000 7f1302b26640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.662+0000 7f1302b26640 1 -- 172.21.15.148:0/4181650145 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc0aa6a0 con 0x7f12fc0b1c10 2023-10-30T12:18:06.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.665+0000 7f1302b26640 1 --2- 172.21.15.148:0/4181650145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 0x7f12fc0b2060 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0b3c70 tx=0x7f12f402bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=970107bbad43d6c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 1 -- 172.21.15.148:0/4181650145 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f402cb40 con 0x7f12fc0b1c10 2023-10-30T12:18:06.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:06.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 10 monclient: dump: 2023-10-30T12:18:06.681 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.681 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.681 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.681 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.682 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.682 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.682 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.682 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.682 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.683 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 1 -- 172.21.15.148:0/4181650145 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f402ccb0 con 0x7f12fc0b1c10 2023-10-30T12:18:06.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 1 -- 172.21.15.148:0/4181650145 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f4034660 con 0x7f12fc0b1c10 2023-10-30T12:18:06.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:06.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1301323640 10 monclient: dump: 2023-10-30T12:18:06.684 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.685 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.685 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.685 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.685 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.685 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.686 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.686 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.686 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.686 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1304589640 5 monclient: authenticate success, global_id 4390 2023-10-30T12:18:06.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1304589640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:06.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.669+0000 7f1304589640 4 set_mon_vals no callback set 2023-10-30T12:18:06.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.670+0000 7f1304589640 10 monclient: shutdown 2023-10-30T12:18:06.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.670+0000 7f1304589640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:06.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.670+0000 7f1304589640 1 -- 172.21.15.148:0/4181650145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 msgr2=0x7f12fc0b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.670+0000 7f1304589640 1 --2- 172.21.15.148:0/4181650145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 0x7f12fc0b2060 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0b3c70 tx=0x7f12f402bd90 comp rx=0 tx=0).stop 2023-10-30T12:18:06.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.671+0000 7f1304589640 1 -- 172.21.15.148:0/4181650145 shutdown_connections 2023-10-30T12:18:06.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.671+0000 7f1304589640 1 --2- 172.21.15.148:0/4181650145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 0x7f12fc0b2060 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.671+0000 7f1304589640 1 --2- 172.21.15.148:0/4181650145 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f12fc141940 0x7f12fc141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.671+0000 7f1304589640 1 --2- 172.21.15.148:0/4181650145 >> v2:172.21.15.148:3300/0 conn(0x7f12fc0b7c10 0x7f12fc0a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.671+0000 7f1304589640 1 -- 172.21.15.148:0/4181650145 >> 172.21.15.148:0/4181650145 conn(0x7f12fc06c660 msgr2=0x7f12fc0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:06.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.671+0000 7f1304589640 1 -- 172.21.15.148:0/4181650145 shutdown_connections 2023-10-30T12:18:06.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.671+0000 7f1304589640 1 -- 172.21.15.148:0/4181650145 wait complete. 2023-10-30T12:18:06.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.672+0000 7f1304589640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.672+0000 7f1304589640 10 monclient: monmap: 2023-10-30T12:18:06.689 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.690 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:06.690 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:06.690 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:06.690 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.690 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:06.690 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:06.691 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:06.691 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:06.691 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.672+0000 7f1304589640 1 librados: starting msgr at 2023-10-30T12:18:06.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.672+0000 7f1304589640 1 librados: starting objecter 2023-10-30T12:18:06.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.673+0000 7f1304589640 1 Processor -- start 2023-10-30T12:18:06.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.673+0000 7f1304589640 1 -- start start 2023-10-30T12:18:06.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.673+0000 7f1304589640 1 librados: setting wanted keys 2023-10-30T12:18:06.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.673+0000 7f1304589640 1 librados: calling monclient init 2023-10-30T12:18:06.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.673+0000 7f1304589640 10 monclient: init 2023-10-30T12:18:06.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:06.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 0x7f12fc06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 10 monclient(hunting): picked mon.noname-b con 0x7f12fc0b1c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:06.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f12fc0b7c10 0x7f12fc0b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 10 monclient(hunting): picked mon.noname-c con 0x7f12fc0b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:06.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12fc141940 0x7f12fc1db470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 10 monclient(hunting): picked mon.noname-a con 0x7f12fc141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:06.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc141ff0 con 0x7f12fc141940 2023-10-30T12:18:06.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.674+0000 7f1304589640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.675+0000 7f1304589640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc06dff0 con 0x7f12fc0b1c10 2023-10-30T12:18:06.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.675+0000 7f1304589640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.675+0000 7f1304589640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc1daf90 con 0x7f12fc0b7c10 2023-10-30T12:18:06.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.675+0000 7f1304589640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.675+0000 7f1304589640 10 monclient(hunting): authenticate will time out at 1578.880135s 2023-10-30T12:18:06.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.676+0000 7f1302b26640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12fc141940 0x7f12fc1db470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302325640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 0x7f12fc06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302325640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 0x7f12fc06d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:48978/0 (socket says 172.21.15.148:48978) 2023-10-30T12:18:06.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302325640 1 -- 172.21.15.148:0/3102636917 learned_addr learned my addr 172.21.15.148:0/3102636917 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:06.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302325640 10 monclient(hunting): get_auth_request con 0x7f12fc0b1c10 auth_method 0 2023-10-30T12:18:06.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302325640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302325640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302325640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302b26640 10 monclient(hunting): get_auth_request con 0x7f12fc141940 auth_method 0 2023-10-30T12:18:06.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302b26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302b26640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302b26640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.677+0000 7f1302325640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.678+0000 7f1302325640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.678+0000 7f1302325640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.679+0000 7f1301b24640 1 --2- 172.21.15.148:0/3102636917 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f12fc0b7c10 0x7f12fc0b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.679+0000 7f1301b24640 10 monclient(hunting): get_auth_request con 0x7f12fc0b7c10 auth_method 0 2023-10-30T12:18:06.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.679+0000 7f1301b24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.679+0000 7f1301b24640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.679+0000 7f1301b24640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.680+0000 7f1301b24640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.680+0000 7f1301b24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.680+0000 7f1301b24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.682+0000 7f1302325640 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2023-10-30T12:18:06.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.682+0000 7f1302325640 1 -- 172.21.15.148:0/3102636917 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f12fc0b7c10 msgr2=0x7f12fc0b0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.682+0000 7f1302325640 1 --2- 172.21.15.148:0/3102636917 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f12fc0b7c10 0x7f12fc0b0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.683+0000 7f1302325640 1 -- 172.21.15.148:0/3102636917 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12fc141940 msgr2=0x7f12fc1db470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.683+0000 7f1302325640 1 --2- 172.21.15.148:0/3102636917 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12fc141940 0x7f12fc1db470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.683+0000 7f1302325640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.683+0000 7f1302325640 1 monclient: found mon.noname-b 2023-10-30T12:18:06.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.683+0000 7f1302325640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.683+0000 7f1302325640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.683+0000 7f1302325640 1 -- 172.21.15.148:0/3102636917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc0adfe0 con 0x7f12fc0b1c10 2023-10-30T12:18:06.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.683+0000 7f1302b26640 1 --2- 172.21.15.148:0/3102636917 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12fc141940 0x7f12fc1db470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:18:06.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.684+0000 7f1302325640 1 --2- 172.21.15.148:0/3102636917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 0x7f12fc06d9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12f800e8b0 tx=0x7f12f800ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.684+0000 7f12f2ffd640 10 client.?.objecter ms_handle_connect 0x7f12fc0b1c10 2023-10-30T12:18:06.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.685+0000 7f12f2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:06.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.688+0000 7f12f2ffd640 1 -- 172.21.15.148:0/3102636917 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f800c940 con 0x7f12fc0b1c10 2023-10-30T12:18:06.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.688+0000 7f12f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.688+0000 7f12f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:06.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.688+0000 7f12f2ffd640 10 monclient: dump: 2023-10-30T12:18:06.707 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.707 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.707 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.707 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.707 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.708 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.708 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.708 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.708 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.708 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.688+0000 7f12f2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f12f2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f12f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f1304589640 5 monclient: authenticate success, global_id 4393 2023-10-30T12:18:06.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f1304589640 10 monclient: _renew_subs 2023-10-30T12:18:06.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f1304589640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f1304589640 1 -- 172.21.15.148:0/3102636917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12fc0afc30 con 0x7f12fc0b1c10 2023-10-30T12:18:06.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f1304589640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:06.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f1304589640 10 monclient: _renew_subs 2023-10-30T12:18:06.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f1304589640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f1304589640 1 -- 172.21.15.148:0/3102636917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12fc1db9b0 con 0x7f12fc0b1c10 2023-10-30T12:18:06.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f1304589640 1 librados: init done 2023-10-30T12:18:06.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f12f2ffd640 1 -- 172.21.15.148:0/3102636917 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f800d040 con 0x7f12fc0b1c10 2023-10-30T12:18:06.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f12f2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.689+0000 7f1300b22640 4 set_mon_vals no callback set 2023-10-30T12:18:06.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.690+0000 7f12f2ffd640 1 -- 172.21.15.148:0/3102636917 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f800ccd0 con 0x7f12fc0b1c10 2023-10-30T12:18:06.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.690+0000 7f12f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.690+0000 7f12f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:06.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.690+0000 7f12f2ffd640 10 monclient: dump: 2023-10-30T12:18:06.712 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.712 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.712 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.713 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.713 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.713 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.713 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.713 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.714 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.714 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.690+0000 7f12f0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.690+0000 7f12f0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.690+0000 7f12f0ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:06.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.690+0000 7f12f0ff9640 1 -- 172.21.15.148:0/3102636917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12fc075550 con 0x7f12fc0b1c10 2023-10-30T12:18:06.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.696+0000 7fd2c4061640 10 monclient: get_monmap_and_config 2023-10-30T12:18:06.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.696+0000 7fd2c4061640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.696+0000 7fd2c4061640 10 monclient: monmap: 2023-10-30T12:18:06.716 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.716 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.716 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:06.697858+0000 2023-10-30T12:18:06.717 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:06.697858+0000 2023-10-30T12:18:06.717 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.717 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.717 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:06.717 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:06.717 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:06.718 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:06.718 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:06.718 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.697+0000 7fd2c4061640 1 Processor -- start 2023-10-30T12:18:06.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.698+0000 7fd2c4061640 1 -- start start 2023-10-30T12:18:06.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.698+0000 7fd2c4061640 10 monclient: init 2023-10-30T12:18:06.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.698+0000 7fe2e2e0e640 10 monclient: get_monmap_and_config 2023-10-30T12:18:06.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.698+0000 7fe2e2e0e640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.698+0000 7fe2e2e0e640 10 monclient: monmap: 2023-10-30T12:18:06.720 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.720 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.720 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:06.699493+0000 2023-10-30T12:18:06.720 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:06.699493+0000 2023-10-30T12:18:06.720 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.721 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.721 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:06.721 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:06.721 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:06.721 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:06.722 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:06.722 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.698+0000 7fd2c4061640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.698+0000 7fd2c4061640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T12:18:06.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.698+0000 7fd2c4061640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2bc0b7c10 0x7fd2bc0a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.698+0000 7fd2c4061640 10 monclient(hunting): picked mon.noname-c con 0x7fd2bc0b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:06.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.698+0000 7fd2c4061640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2bc141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:06.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fd2c4061640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd2bc13e730 0x7fd2bc13eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fd2c4061640 10 monclient(hunting): picked mon.noname-a con 0x7fd2bc13e730 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:06.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fd2c4061640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc0a9350 con 0x7fd2bc141940 2023-10-30T12:18:06.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fd2c4061640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fd2c4061640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc0670a0 con 0x7fd2bc13e730 2023-10-30T12:18:06.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fd2c4061640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fd2c4061640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc0b7ff0 con 0x7fd2bc0b7c10 2023-10-30T12:18:06.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fd2c4061640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fd2c4061640 10 monclient(hunting): authenticate will time out at 1578.904244s 2023-10-30T12:18:06.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fe2e2e0e640 1 Processor -- start 2023-10-30T12:18:06.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fe2e2e0e640 1 -- start start 2023-10-30T12:18:06.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.699+0000 7fe2e2e0e640 10 monclient: init 2023-10-30T12:18:06.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-10-30T12:18:06.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc0b7c10 0x7fe2dc0a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 10 monclient(hunting): picked mon.noname-b con 0x7fe2dc0b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:06.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2dc141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:06.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe2dc13e730 0x7fe2dc13eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 10 monclient(hunting): picked mon.noname-c con 0x7fe2dc13e730 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:06.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc0a9350 con 0x7fe2dc141940 2023-10-30T12:18:06.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc0670a0 con 0x7fe2dc13e730 2023-10-30T12:18:06.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc142470 con 0x7fe2dc0b7c10 2023-10-30T12:18:06.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.700+0000 7fe2e2e0e640 10 monclient(hunting): authenticate will time out at 1578.905197s 2023-10-30T12:18:06.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.703+0000 7fc9e5d9b640 1 -- 172.21.15.148:0/2617958551 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e002ca10 con 0x7fc9e80b1c10 2023-10-30T12:18:06.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.703+0000 7fc9e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.703+0000 7fc9e5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:06.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.703+0000 7fc9e5d9b640 10 monclient: dump: 2023-10-30T12:18:06.733 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.733 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.733 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.734 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.734 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.734 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.734 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.734 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.734 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.735 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.703+0000 7fc9e5d9b640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.703+0000 7fc9e5d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.703+0000 7fc9e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.703+0000 7fc9e5d9b640 1 -- 172.21.15.148:0/2617958551 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e002cb80 con 0x7fc9e80b1c10 2023-10-30T12:18:06.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.703+0000 7fc9ecdbb640 5 monclient: authenticate success, global_id 4301 2023-10-30T12:18:06.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9ecdbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:06.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9e5d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9ecdbb640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:06.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9ecdbb640 4 set_mon_vals no callback set 2023-10-30T12:18:06.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9e5d9b640 1 -- 172.21.15.148:0/2617958551 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e00343f0 con 0x7fc9e80b1c10 2023-10-30T12:18:06.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9e5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:06.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9e5d9b640 10 monclient: dump: 2023-10-30T12:18:06.737 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.737 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.738 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.738 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.738 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.738 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.738 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.739 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.739 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.739 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9ecdbb640 10 monclient: shutdown 2023-10-30T12:18:06.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9ecdbb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:06.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/2617958551 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b1c10 msgr2=0x7fc9e80b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.704+0000 7fc9ecdbb640 1 --2- 172.21.15.148:0/2617958551 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b1c10 0x7fc9e80b2060 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80b4d80 tx=0x7fc9e002be70 comp rx=0 tx=0).stop 2023-10-30T12:18:06.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.705+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/2617958551 shutdown_connections 2023-10-30T12:18:06.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.705+0000 7fc9ecdbb640 1 --2- 172.21.15.148:0/2617958551 >> v2:172.21.15.148:3300/0 conn(0x7fc9e8141940 0x7fc9e8141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.705+0000 7fc9ecdbb640 1 --2- 172.21.15.148:0/2617958551 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b1c10 0x7fc9e80b2060 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.705+0000 7fc9ecdbb640 1 --2- 172.21.15.148:0/2617958551 >> v2:172.21.15.150:3300/0 conn(0x7fc9e80b7c10 0x7fc9e80a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.705+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/2617958551 >> 172.21.15.148:0/2617958551 conn(0x7fc9e806c660 msgr2=0x7fc9e80b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:06.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.705+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/2617958551 shutdown_connections 2023-10-30T12:18:06.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.705+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/2617958551 wait complete. 2023-10-30T12:18:06.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.705+0000 7fc9ecdbb640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.705+0000 7fc9ecdbb640 10 monclient: monmap: 2023-10-30T12:18:06.741 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.742 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:06.742 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:06.742 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:06.742 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.742 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:06.743 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:06.743 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:06.743 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:06.743 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 librados: starting msgr at 2023-10-30T12:18:06.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 librados: starting objecter 2023-10-30T12:18:06.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 Processor -- start 2023-10-30T12:18:06.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 -- start start 2023-10-30T12:18:06.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 librados: setting wanted keys 2023-10-30T12:18:06.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 librados: calling monclient init 2023-10-30T12:18:06.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient: init 2023-10-30T12:18:06.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:06.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e80b1c10 0x7fc9e81d6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e80b1c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:06.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b7c10 0x7fc9e811f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient(hunting): picked mon.noname-c con 0x7fc9e80b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:06.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e8141940 0x7fc9e8139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient(hunting): picked mon.noname-a con 0x7fc9e8141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:06.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e8141fe0 con 0x7fc9e8141940 2023-10-30T12:18:06.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e813a880 con 0x7fc9e80b1c10 2023-10-30T12:18:06.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e80b0130 con 0x7fc9e80b7c10 2023-10-30T12:18:06.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.706+0000 7fc9ecdbb640 10 monclient(hunting): authenticate will time out at 1578.911808s 2023-10-30T12:18:06.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.709+0000 7f12f2ffd640 1 -- 172.21.15.148:0/3102636917 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f12f8013470 con 0x7f12fc0b1c10 2023-10-30T12:18:06.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.709+0000 7f12f2ffd640 1 --2- 172.21.15.148:0/3102636917 >> v2:172.21.15.148:6800/3205090179 conn(0x7f12e006c810 0x7f12e006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.710+0000 7f1301b24640 1 --2- 172.21.15.148:0/3102636917 >> v2:172.21.15.148:6800/3205090179 conn(0x7f12e006c810 0x7f12e006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.710+0000 7f1301b24640 10 monclient: get_auth_request con 0x7f12e006c810 auth_method 0 2023-10-30T12:18:06.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.710+0000 7f7a56661640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:18:06.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.710+0000 7f7a56661640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:18:06.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.710+0000 7f7a56661640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.710+0000 7f7a56661640 1 -- 172.21.15.148:0/236113376 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7a501da870 con 0x7f7a500b7c10 2023-10-30T12:18:06.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.710+0000 7f12f2ffd640 10 client.4393.objecter ms_handle_connect 0x7f12e006c810 2023-10-30T12:18:06.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.710+0000 7f1301b24640 1 --2- 172.21.15.148:0/3102636917 >> v2:172.21.15.148:6800/3205090179 conn(0x7f12e006c810 0x7f12e006ecd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0b1080 tx=0x7f12ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.711+0000 7f12f2ffd640 1 -- 172.21.15.148:0/3102636917 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f12f80586f0 con 0x7f12fc0b1c10 2023-10-30T12:18:06.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.711+0000 7f12f2ffd640 10 client.4393.objecter ms_dispatch 0x7f12fc000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:06.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.711+0000 7f12f2ffd640 3 client.4393.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:06.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.711+0000 7f12f2ffd640 3 client.4393.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:06.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.711+0000 7f12f2ffd640 20 client.4393.objecter dump_active .. 0 homeless 2023-10-30T12:18:06.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.745+0000 7f7a4cff9640 1 -- 172.21.15.148:0/236113376 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7a4005b8a0 con 0x7f7a500b7c10 2023-10-30T12:18:06.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.745+0000 7f7a4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:18:06.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.745+0000 7f7a4cff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:06.753 INFO:teuthology.orchestra.run.smithi148.stdout:64424509445 2023-10-30T12:18:06.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.746+0000 7fd2c25fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd2bc13e730 0x7fd2bc13eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 10 librados: watch_flush enter 2023-10-30T12:18:06.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 10 librados: watch_flush exit 2023-10-30T12:18:06.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 20 client.4344.objecter shutdown clearing up homeless session... 2023-10-30T12:18:06.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 10 client.4344.objecter successfully canceled tick 2023-10-30T12:18:06.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7fc9e6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e8141940 0x7fc9e8139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 -- 172.21.15.148:0/236113376 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7a3c06c7c0 msgr2=0x7f7a3c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 --2- 172.21.15.148:0/236113376 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7a3c06c7c0 0x7f7a3c06ec80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7a44036000 tx=0x7f7a44030000 comp rx=0 tx=0).stop 2023-10-30T12:18:06.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 10 monclient: shutdown 2023-10-30T12:18:06.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:06.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 -- 172.21.15.148:0/236113376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a500b7c10 msgr2=0x7f7a501da300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 --2- 172.21.15.148:0/236113376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a500b7c10 0x7f7a501da300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000c450 tx=0x7f7a4000c910 comp rx=0 tx=0).stop 2023-10-30T12:18:06.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 -- 172.21.15.148:0/236113376 shutdown_connections 2023-10-30T12:18:06.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 --2- 172.21.15.148:0/236113376 >> v2:172.21.15.148:6800/3205090179 conn(0x7f7a3c06c7c0 0x7f7a3c06ec80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 --2- 172.21.15.148:0/236113376 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7a50141940 0x7f7a500ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 --2- 172.21.15.148:0/236113376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a5013e730 0x7f7a500ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 --2- 172.21.15.148:0/236113376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a500b7c10 0x7f7a501da300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 -- 172.21.15.148:0/236113376 >> 172.21.15.148:0/236113376 conn(0x7f7a5006c660 msgr2=0x7f7a50142450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:06.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 -- 172.21.15.148:0/236113376 shutdown_connections 2023-10-30T12:18:06.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 -- 172.21.15.148:0/236113376 wait complete. 2023-10-30T12:18:06.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7f7a56661640 1 librados: shutdown 2023-10-30T12:18:06.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7fd2c25fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd2bc13e730 0x7fd2bc13eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60116/0 (socket says 172.21.15.148:60116) 2023-10-30T12:18:06.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7fd2c25fe640 1 -- 172.21.15.148:0/1316779675 learned_addr learned my addr 172.21.15.148:0/1316779675 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:06.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7fd2c25fe640 10 monclient(hunting): get_auth_request con 0x7fd2bc13e730 auth_method 0 2023-10-30T12:18:06.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7fd2c25fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7fd2c25fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7fd2c25fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.747+0000 7fc9e6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e8141940 0x7fc9e8139bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60132/0 (socket says 172.21.15.148:60132) 2023-10-30T12:18:06.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.748+0000 7fc9e6d9d640 1 -- 172.21.15.148:0/4054374812 learned_addr learned my addr 172.21.15.148:0/4054374812 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:06.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.748+0000 7fc9e6d9d640 10 monclient(hunting): get_auth_request con 0x7fc9e8141940 auth_method 0 2023-10-30T12:18:06.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.748+0000 7fc9e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.748+0000 7fc9e6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.748+0000 7fc9e6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.749+0000 7fd2c25fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.749+0000 7fd2c25fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fd2c25fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fe2e0baa640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc0b7c10 0x7fe2dc0a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fc9e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fc9e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fc9e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fe2e0baa640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc0b7c10 0x7fe2dc0a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54648/0 (socket says 172.21.15.148:54648) 2023-10-30T12:18:06.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fe2e0baa640 1 -- 172.21.15.148:0/4150312891 learned_addr learned my addr 172.21.15.148:0/4150312891 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:06.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fe2e0baa640 10 monclient(hunting): get_auth_request con 0x7fe2dc0b7c10 auth_method 0 2023-10-30T12:18:06.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fe2e0baa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fe2e0baa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.750+0000 7fe2e0baa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.751+0000 7fc9dffff640 1 --2- 172.21.15.148:0/4054374812 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b7c10 0x7fc9e811f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.751+0000 7fe2e0baa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.751+0000 7fe2e0baa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.751+0000 7fe2e0baa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fd2c25fe640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2023-10-30T12:18:06.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fd2c25fe640 1 -- 172.21.15.148:0/1316779675 >> v2:172.21.15.150:3300/0 conn(0x7fd2bc0b7c10 msgr2=0x7fd2bc0a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:06.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fd2c25fe640 1 --2- 172.21.15.148:0/1316779675 >> v2:172.21.15.150:3300/0 conn(0x7fd2bc0b7c10 0x7fd2bc0a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fd2c25fe640 1 -- 172.21.15.148:0/1316779675 >> v1:172.21.15.150:6789/0 conn(0x7fd2bc141940 legacy=0x7fd2bc141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:06.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fd2c25fe640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fd2c25fe640 1 monclient: found mon.noname-a 2023-10-30T12:18:06.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fd2c25fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fd2c25fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fd2c25fe640 1 -- 172.21.15.148:0/1316779675 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc0aa6a0 con 0x7fd2bc13e730 2023-10-30T12:18:06.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fc9dffff640 10 monclient(hunting): get_auth_request con 0x7fc9e80b7c10 auth_method 0 2023-10-30T12:18:06.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fc9dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fc9dffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.752+0000 7fc9dffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.754+0000 7fe2e0baa640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2023-10-30T12:18:06.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.754+0000 7fe2e0baa640 1 -- 172.21.15.148:0/4150312891 >> v2:172.21.15.150:3300/0 conn(0x7fe2dc13e730 msgr2=0x7fe2dc13eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:06.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.754+0000 7fe2e0baa640 1 --2- 172.21.15.148:0/4150312891 >> v2:172.21.15.150:3300/0 conn(0x7fe2dc13e730 0x7fe2dc13eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.754+0000 7fe2e0baa640 1 -- 172.21.15.148:0/4150312891 >> v1:172.21.15.150:6789/0 conn(0x7fe2dc141940 legacy=0x7fe2dc141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:06.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.754+0000 7fe2e0baa640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.754+0000 7fe2e0baa640 1 monclient: found mon.noname-b 2023-10-30T12:18:06.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.754+0000 7fe2e0baa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.754+0000 7fe2e0baa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:18:06.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.754+0000 7fe2e0baa640 1 -- 172.21.15.148:0/4150312891 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0aa6a0 con 0x7fe2dc0b7c10 2023-10-30T12:18:06.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9e6d9d640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2023-10-30T12:18:06.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9e6d9d640 1 -- 172.21.15.148:0/4054374812 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b7c10 msgr2=0x7fc9e811f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9e6d9d640 1 --2- 172.21.15.148:0/4054374812 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b7c10 0x7fc9e811f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9e6d9d640 1 -- 172.21.15.148:0/4054374812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e80b1c10 msgr2=0x7fc9e81d6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9e6d9d640 1 --2- 172.21.15.148:0/4054374812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e80b1c10 0x7fc9e81d6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9e6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9e6d9d640 1 monclient: found mon.noname-a 2023-10-30T12:18:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9e6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.756+0000 7fc9e6d9d640 1 -- 172.21.15.148:0/4054374812 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e80b0d70 con 0x7fc9e8141940 2023-10-30T12:18:06.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.0 2023-10-30T12:18:06.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.761+0000 7fd2c25fe640 1 --2- 172.21.15.148:0/1316779675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2bc13e730 0x7fd2bc13eb80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0b4d80 tx=0x7fd2b402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9158eff686bd594 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.777 DEBUG:teuthology.parallel:result is None 2023-10-30T12:18:06.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.761+0000 7fc9e6d9d640 1 --2- 172.21.15.148:0/4054374812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e8141940 0x7fc9e8139bf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0036040 tx=0x7fc9e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.761+0000 7fc9deffd640 10 client.?.objecter ms_handle_connect 0x7fc9e8141940 2023-10-30T12:18:06.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.761+0000 7fc9deffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:06.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.763+0000 7fe2e0baa640 1 --2- 172.21.15.148:0/4150312891 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc0b7c10 0x7fe2dc0a8aa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009870 tx=0x7fe2d002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6a1c90ff4ab0d55 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.767+0000 7fc9deffd640 1 -- 172.21.15.148:0/4054374812 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e00094d0 con 0x7fc9e8141940 2023-10-30T12:18:06.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.767+0000 7fc9deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.767+0000 7fc9deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:06.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.767+0000 7fc9deffd640 10 monclient: dump: 2023-10-30T12:18:06.780 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.780 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.780 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.780 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.780 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.781 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.781 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.781 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.781 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.781 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9deffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9deffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9deffd640 1 -- 172.21.15.148:0/4054374812 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e003e450 con 0x7fc9e8141940 2023-10-30T12:18:06.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9ecdbb640 5 monclient: authenticate success, global_id 4356 2023-10-30T12:18:06.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9ecdbb640 10 monclient: _renew_subs 2023-10-30T12:18:06.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9ecdbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/4054374812 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e81d7280 con 0x7fc9e8141940 2023-10-30T12:18:06.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9deffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9deffd640 1 -- 172.21.15.148:0/4054374812 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e003e850 con 0x7fc9e8141940 2023-10-30T12:18:06.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:06.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9deffd640 10 monclient: dump: 2023-10-30T12:18:06.784 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.784 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.784 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.785 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.785 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.785 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.785 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.785 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.785 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.786 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9e559a640 4 set_mon_vals no callback set 2023-10-30T12:18:06.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9ecdbb640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:06.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9ecdbb640 10 monclient: _renew_subs 2023-10-30T12:18:06.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9ecdbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/4054374812 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e811f190 con 0x7fc9e8141940 2023-10-30T12:18:06.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fc9ecdbb640 1 librados: init done 2023-10-30T12:18:06.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 1 -- 172.21.15.148:0/4150312891 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d00317c0 con 0x7fe2dc0b7c10 2023-10-30T12:18:06.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:06.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 10 monclient: dump: 2023-10-30T12:18:06.788 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.788 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.789 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.789 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.789 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.789 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.789 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.789 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.790 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.790 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 1 -- 172.21.15.148:0/4150312891 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d002e070 con 0x7fe2dc0b7c10 2023-10-30T12:18:06.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2e2e0e640 5 monclient: authenticate success, global_id 4310 2023-10-30T12:18:06.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2e2e0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:06.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2e2e0e640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:06.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2e2e0e640 4 set_mon_vals no callback set 2023-10-30T12:18:06.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 1 -- 172.21.15.148:0/4150312891 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0031b50 con 0x7fe2dc0b7c10 2023-10-30T12:18:06.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:06.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.768+0000 7fe2db7fe640 10 monclient: dump: 2023-10-30T12:18:06.792 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.793 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.793 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.793 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.793 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.793 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.794 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.794 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.794 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.794 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fc9ecdbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fc9ecdbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fc9ecdbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:06.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/4054374812 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e8075550 con 0x7fc9e8141940 2023-10-30T12:18:06.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fe2e2e0e640 10 monclient: shutdown 2023-10-30T12:18:06.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fe2e2e0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:06.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/4150312891 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc0b7c10 msgr2=0x7fe2dc0a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fe2e2e0e640 1 --2- 172.21.15.148:0/4150312891 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc0b7c10 0x7fe2dc0a8aa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009870 tx=0x7fe2d002b860 comp rx=0 tx=0).stop 2023-10-30T12:18:06.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/4150312891 shutdown_connections 2023-10-30T12:18:06.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fe2e2e0e640 1 --2- 172.21.15.148:0/4150312891 >> v2:172.21.15.150:3300/0 conn(0x7fe2dc13e730 0x7fe2dc13eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fe2e2e0e640 1 --2- 172.21.15.148:0/4150312891 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc0b7c10 0x7fe2dc0a8aa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/4150312891 >> 172.21.15.148:0/4150312891 conn(0x7fe2dc06c660 msgr2=0x7fe2dc0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:06.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/4150312891 shutdown_connections 2023-10-30T12:18:06.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.769+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/4150312891 wait complete. 2023-10-30T12:18:06.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.770+0000 7fe2e2e0e640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.770+0000 7fe2e2e0e640 10 monclient: monmap: 2023-10-30T12:18:06.801 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.801 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:06.801 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:06.801 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:06.802 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.802 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:06.802 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:06.802 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:06.802 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:06.803 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.770+0000 7fe2e2e0e640 1 librados: starting msgr at 2023-10-30T12:18:06.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.770+0000 7fe2e2e0e640 1 librados: starting objecter 2023-10-30T12:18:06.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.770+0000 7fe2e2e0e640 1 Processor -- start 2023-10-30T12:18:06.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.770+0000 7fe2e2e0e640 1 -- start start 2023-10-30T12:18:06.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.770+0000 7fe2e2e0e640 1 librados: setting wanted keys 2023-10-30T12:18:06.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.770+0000 7fe2e2e0e640 1 librados: calling monclient init 2023-10-30T12:18:06.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.770+0000 7fe2e2e0e640 10 monclient: init 2023-10-30T12:18:06.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:06.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2dc0b7c10 0x7fe2dc1dadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 10 monclient(hunting): picked mon.noname-b con 0x7fe2dc0b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:06.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc13e730 0x7fe2dc13a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 10 monclient(hunting): picked mon.noname-c con 0x7fe2dc13e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:06.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2dc141940 0x7fe2dc1da4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 10 monclient(hunting): picked mon.noname-a con 0x7fe2dc141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:06.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0b2fd0 con 0x7fe2dc141940 2023-10-30T12:18:06.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0afa00 con 0x7fe2dc0b7c10 2023-10-30T12:18:06.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0b0170 con 0x7fe2dc13e730 2023-10-30T12:18:06.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.771+0000 7fe2e2e0e640 10 monclient(hunting): authenticate will time out at 1578.976552s 2023-10-30T12:18:06.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.772+0000 7fe2dbfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc13e730 0x7fe2dc13a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.772+0000 7fe2dbfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc13e730 0x7fe2dc13a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54668/0 (socket says 172.21.15.148:54668) 2023-10-30T12:18:06.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.772+0000 7fe2dbfff640 1 -- 172.21.15.148:0/2463144324 learned_addr learned my addr 172.21.15.148:0/2463144324 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:06.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.772+0000 7fe2dbfff640 10 monclient(hunting): get_auth_request con 0x7fe2dc13e730 auth_method 0 2023-10-30T12:18:06.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.772+0000 7fe2dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.772+0000 7fe2dbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.772+0000 7fe2dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.772+0000 7fe2dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.772+0000 7fe2dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.772+0000 7fe2dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:06.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.776+0000 7fe2dbfff640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2023-10-30T12:18:06.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.776+0000 7fe2dbfff640 1 -- 172.21.15.148:0/2463144324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2dc0b7c10 msgr2=0x7fe2dc1dadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.776+0000 7fe2dbfff640 1 --2- 172.21.15.148:0/2463144324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2dc0b7c10 0x7fe2dc1dadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.776+0000 7fe2dbfff640 1 -- 172.21.15.148:0/2463144324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2dc141940 msgr2=0x7fe2dc1da4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.776+0000 7fe2dbfff640 1 --2- 172.21.15.148:0/2463144324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2dc141940 0x7fe2dc1da4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.776+0000 7fe2dbfff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:06.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.776+0000 7fe2dbfff640 1 monclient: found mon.noname-c 2023-10-30T12:18:06.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.776+0000 7fe2dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:06.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.776+0000 7fe2dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:18:06.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.776+0000 7fe2dbfff640 1 -- 172.21.15.148:0/2463144324 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0b0db0 con 0x7fe2dc13e730 2023-10-30T12:18:06.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.778+0000 7fe2dbfff640 1 --2- 172.21.15.148:0/2463144324 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc13e730 0x7fe2dc13a750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800b410 tx=0x7fe2c800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.778+0000 7fe2d97fa640 10 client.?.objecter ms_handle_connect 0x7fe2dc13e730 2023-10-30T12:18:06.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.778+0000 7fe2d97fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:06.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 1 -- 172.21.15.148:0/2463144324 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8011830 con 0x7fe2dc13e730 2023-10-30T12:18:06.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:06.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 10 monclient: dump: 2023-10-30T12:18:06.812 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.812 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.812 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.814 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 1 -- 172.21.15.148:0/2463144324 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c800d040 con 0x7fe2dc13e730 2023-10-30T12:18:06.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 1 -- 172.21.15.148:0/2463144324 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8011be0 con 0x7fe2dc13e730 2023-10-30T12:18:06.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:06.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2d97fa640 10 monclient: dump: 2023-10-30T12:18:06.815 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.816 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.816 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.816 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.816 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.816 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.817 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.817 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.817 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.817 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2e2e0e640 5 monclient: authenticate success, global_id 4316 2023-10-30T12:18:06.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2e2e0e640 10 monclient: _renew_subs 2023-10-30T12:18:06.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2daffd640 4 set_mon_vals no callback set 2023-10-30T12:18:06.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2e2e0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:06.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/2463144324 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc11eed0 con 0x7fe2dc13e730 2023-10-30T12:18:06.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2e2e0e640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:06.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2e2e0e640 10 monclient: _renew_subs 2023-10-30T12:18:06.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2e2e0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:06.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/2463144324 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc11f3a0 con 0x7fe2dc13e730 2023-10-30T12:18:06.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.781+0000 7fe2e2e0e640 1 librados: init done 2023-10-30T12:18:06.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.783+0000 7fe2e2e0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.783+0000 7fe2e2e0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.783+0000 7fe2e2e0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:06.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.783+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/2463144324 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc0a97c0 con 0x7fe2dc13e730 2023-10-30T12:18:06.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.802+0000 7fe2d97fa640 1 -- 172.21.15.148:0/2463144324 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fe2c8011d50 con 0x7fe2dc13e730 2023-10-30T12:18:06.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.802+0000 7fe2d97fa640 1 --2- 172.21.15.148:0/2463144324 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe2c006c7f0 0x7fe2c006ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.802+0000 7fe2e0baa640 1 --2- 172.21.15.148:0/2463144324 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe2c006c7f0 0x7fe2c006ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.802+0000 7fe2e0baa640 10 monclient: get_auth_request con 0x7fe2c006c7f0 auth_method 0 2023-10-30T12:18:06.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.802+0000 7fe2d97fa640 10 client.4316.objecter ms_handle_connect 0x7fe2c006c7f0 2023-10-30T12:18:06.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.802+0000 7fe2e0baa640 1 --2- 172.21.15.148:0/2463144324 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe2c006c7f0 0x7fe2c006ecb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009870 tx=0x7fe2d0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:06.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c0dfb640 1 -- 172.21.15.148:0/1316779675 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b402c9f0 con 0x7fd2bc13e730 2023-10-30T12:18:06.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c0dfb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:06.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c0dfb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:06.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c0dfb640 10 monclient: dump: 2023-10-30T12:18:06.976 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:06.976 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:06.977 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.977 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:06.977 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:06.977 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:06.977 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:06.978 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:06.978 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:06.978 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c0dfb640 10 monclient: _finish_auth 0 2023-10-30T12:18:06.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c0dfb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:06.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c0dfb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:06.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c0dfb640 1 -- 172.21.15.148:0/1316779675 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b402cb60 con 0x7fd2bc13e730 2023-10-30T12:18:06.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 5 monclient: authenticate success, global_id 4353 2023-10-30T12:18:06.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:06.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c0dfb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:06.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:06.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 4 set_mon_vals no callback set 2023-10-30T12:18:06.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 10 monclient: shutdown 2023-10-30T12:18:06.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:06.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 1 -- 172.21.15.148:0/1316779675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2bc13e730 msgr2=0x7fd2bc13eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:06.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 1 --2- 172.21.15.148:0/1316779675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2bc13e730 0x7fd2bc13eb80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0b4d80 tx=0x7fd2b402be50 comp rx=0 tx=0).stop 2023-10-30T12:18:06.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 1 -- 172.21.15.148:0/1316779675 shutdown_connections 2023-10-30T12:18:06.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 1 --2- 172.21.15.148:0/1316779675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2bc13e730 0x7fd2bc13eb80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 1 --2- 172.21.15.148:0/1316779675 >> v2:172.21.15.150:3300/0 conn(0x7fd2bc0b7c10 0x7fd2bc0a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:06.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.974+0000 7fd2c4061640 1 -- 172.21.15.148:0/1316779675 >> 172.21.15.148:0/1316779675 conn(0x7fd2bc06c660 msgr2=0x7fd2bc0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:06.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.975+0000 7fd2c4061640 1 -- 172.21.15.148:0/1316779675 shutdown_connections 2023-10-30T12:18:06.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.975+0000 7fd2c4061640 1 -- 172.21.15.148:0/1316779675 wait complete. 2023-10-30T12:18:06.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.975+0000 7fd2c4061640 10 monclient: build_initial_monmap 2023-10-30T12:18:06.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.975+0000 7fd2c4061640 10 monclient: monmap: 2023-10-30T12:18:06.984 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:06.984 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:06.984 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:06.985 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:06.985 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:06.985 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:06.986 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:06.986 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:06.987 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:06.987 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:06.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.975+0000 7fd2c4061640 1 librados: starting msgr at 2023-10-30T12:18:06.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.975+0000 7fd2c4061640 1 librados: starting objecter 2023-10-30T12:18:06.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 1 Processor -- start 2023-10-30T12:18:06.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 1 -- start start 2023-10-30T12:18:06.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 1 librados: setting wanted keys 2023-10-30T12:18:06.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 1 librados: calling monclient init 2023-10-30T12:18:06.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient: init 2023-10-30T12:18:06.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:06.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:06.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd2bc0b7c10 0x7fd2bc1da4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient(hunting): picked mon.noname-c con 0x7fd2bc0b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:06.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2bc13e730 0x7fd2bc137d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient(hunting): picked mon.noname-a con 0x7fd2bc13e730 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:06.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2bc141940 0x7fd2bc1382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:06.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient(hunting): picked mon.noname-b con 0x7fd2bc141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc13f490 con 0x7fd2bc13e730 2023-10-30T12:18:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc1dae60 con 0x7fd2bc141940 2023-10-30T12:18:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc1387e0 con 0x7fd2bc0b7c10 2023-10-30T12:18:06.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:06.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.976+0000 7fd2c4061640 10 monclient(hunting): authenticate will time out at 1579.181541s 2023-10-30T12:18:06.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.989+0000 7f12f2ffd640 1 -- 172.21.15.148:0/3102636917 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f12f805c790 con 0x7f12fc0b1c10 2023-10-30T12:18:06.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.989+0000 7f12f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:06.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.989+0000 7f12f2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:06.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.994+0000 7fd2c25fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2bc141940 0x7fd2bc1382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:06.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.995+0000 7fd2c25fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2bc141940 0x7fd2bc1382a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:49034/0 (socket says 172.21.15.148:49034) 2023-10-30T12:18:06.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.995+0000 7fd2c25fe640 1 -- 172.21.15.148:0/69133774 learned_addr learned my addr 172.21.15.148:0/69133774 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:06.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.995+0000 7fd2c25fe640 10 monclient(hunting): get_auth_request con 0x7fd2bc141940 auth_method 0 2023-10-30T12:18:06.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.995+0000 7fd2c25fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:06.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.995+0000 7fd2c25fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:06.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.995+0000 7fd2c25fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:06.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.996+0000 7fd2c25fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:06.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.996+0000 7fd2c25fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:06.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:06.996+0000 7fd2c25fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:07.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.003+0000 7fd2c25fe640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2023-10-30T12:18:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.003+0000 7fd2c25fe640 1 -- 172.21.15.148:0/69133774 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd2bc0b7c10 msgr2=0x7fd2bc1da4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.003+0000 7fd2c25fe640 1 --2- 172.21.15.148:0/69133774 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd2bc0b7c10 0x7fd2bc1da4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.003+0000 7fd2c25fe640 1 -- 172.21.15.148:0/69133774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2bc13e730 msgr2=0x7fd2bc137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.003+0000 7fd2c25fe640 1 --2- 172.21.15.148:0/69133774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2bc13e730 0x7fd2bc137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.003+0000 7fd2c25fe640 10 monclient: _finish_hunting 0 2023-10-30T12:18:07.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.003+0000 7fd2c25fe640 1 monclient: found mon.noname-b 2023-10-30T12:18:07.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.003+0000 7fd2c25fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:07.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.003+0000 7fd2c25fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.003+0000 7fd2c25fe640 1 -- 172.21.15.148:0/69133774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc0ae480 con 0x7fd2bc141940 2023-10-30T12:18:07.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.005+0000 7fd2c25fe640 1 --2- 172.21.15.148:0/69133774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2bc141940 0x7fd2bc1382a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40093e0 tx=0x7fd2b4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.005+0000 7fd2b27fc640 10 client.?.objecter ms_handle_connect 0x7fd2bc141940 2023-10-30T12:18:07.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.005+0000 7fd2b27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:07.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 1 -- 172.21.15.148:0/69133774 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4009510 con 0x7fd2bc141940 2023-10-30T12:18:07.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:07.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 10 monclient: dump: 2023-10-30T12:18:07.015 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.015 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.015 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.015 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.016 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.016 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.016 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.016 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.016 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.017 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:07.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:07.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:07.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 1 -- 172.21.15.148:0/69133774 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4004250 con 0x7fd2bc141940 2023-10-30T12:18:07.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2c4061640 5 monclient: authenticate success, global_id 4396 2023-10-30T12:18:07.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2c4061640 10 monclient: _renew_subs 2023-10-30T12:18:07.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2c4061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2c4061640 1 -- 172.21.15.148:0/69133774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc1da9f0 con 0x7fd2bc141940 2023-10-30T12:18:07.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:07.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 1 -- 172.21.15.148:0/69133774 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4004610 con 0x7fd2bc141940 2023-10-30T12:18:07.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:07.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b27fc640 10 monclient: dump: 2023-10-30T12:18:07.020 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.020 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.020 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.020 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.021 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.021 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.021 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.021 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.021 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.022 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2b3fff640 4 set_mon_vals no callback set 2023-10-30T12:18:07.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2c4061640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:07.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2c4061640 10 monclient: _renew_subs 2023-10-30T12:18:07.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2c4061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2c4061640 1 -- 172.21.15.148:0/69133774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc138c30 con 0x7fd2bc141940 2023-10-30T12:18:07.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.009+0000 7fd2c4061640 1 librados: init done 2023-10-30T12:18:07.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.010+0000 7fd2c4061640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.010+0000 7fd2c4061640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.010+0000 7fd2c4061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.010+0000 7fd2c4061640 1 -- 172.21.15.148:0/69133774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc138dd0 con 0x7fd2bc141940 2023-10-30T12:18:07.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.028+0000 7fc9deffd640 1 -- 172.21.15.148:0/4054374812 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fc9e002ca20 con 0x7fc9e8141940 2023-10-30T12:18:07.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.029+0000 7fc9deffd640 1 --2- 172.21.15.148:0/4054374812 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc9cc06c7c0 0x7fc9cc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.029+0000 7fc9e659c640 1 --2- 172.21.15.148:0/4054374812 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc9cc06c7c0 0x7fc9cc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.029+0000 7fc9e659c640 10 monclient: get_auth_request con 0x7fc9cc06c7c0 auth_method 0 2023-10-30T12:18:07.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.029+0000 7fc9e659c640 1 --2- 172.21.15.148:0/4054374812 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc9cc06c7c0 0x7fc9cc06ec80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc9d80098f0 tx=0x7fc9d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.029+0000 7fc9deffd640 10 client.4356.objecter ms_handle_connect 0x7fc9cc06c7c0 2023-10-30T12:18:07.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.042+0000 7fc9deffd640 1 -- 172.21.15.148:0/4054374812 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fc9e007ae50 con 0x7fc9e8141940 2023-10-30T12:18:07.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.042+0000 7fc9deffd640 10 client.4356.objecter ms_dispatch 0x7fc9e8000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:07.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.042+0000 7fc9deffd640 3 client.4356.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:07.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.042+0000 7fc9deffd640 3 client.4356.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:07.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.042+0000 7fc9deffd640 20 client.4356.objecter dump_active .. 0 homeless 2023-10-30T12:18:07.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.042+0000 7fc9deffd640 1 -- 172.21.15.148:0/4054374812 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc9e007a750 con 0x7fc9e8141940 2023-10-30T12:18:07.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.042+0000 7fc9deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.042+0000 7fc9deffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:07.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.049+0000 7fe2d97fa640 1 -- 172.21.15.148:0/2463144324 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fe2c8022390 con 0x7fe2dc13e730 2023-10-30T12:18:07.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.049+0000 7fe2d97fa640 10 client.4316.objecter ms_dispatch 0x7fe2dc000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:07.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.049+0000 7fe2d97fa640 3 client.4316.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:07.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.049+0000 7fe2d97fa640 3 client.4316.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:07.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.049+0000 7fe2d97fa640 20 client.4316.objecter dump_active .. 0 homeless 2023-10-30T12:18:07.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.049+0000 7fe2d97fa640 1 -- 172.21.15.148:0/2463144324 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2c8058590 con 0x7fe2dc13e730 2023-10-30T12:18:07.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.049+0000 7fe2d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.049+0000 7fe2d97fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:07.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.188+0000 7f12f0ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:18:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.188+0000 7f12f0ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:18:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.188+0000 7f12f0ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.188+0000 7f12f0ff9640 1 -- 172.21.15.148:0/3102636917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f12fc0adfe0 con 0x7f12fc0b1c10 2023-10-30T12:18:07.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.233+0000 7fc9ecdbb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:18:07.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.233+0000 7fc9ecdbb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:18:07.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.233+0000 7fc9ecdbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:18:07.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.233+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/4054374812 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc9e80aa6a0 con 0x7fc9e8141940 2023-10-30T12:18:07.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.236+0000 7fe2e2e0e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:18:07.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.236+0000 7fe2e2e0e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:18:07.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.236+0000 7fe2e2e0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:07.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.236+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/2463144324 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe2dc11eed0 con 0x7fe2dc13e730 2023-10-30T12:18:07.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.236+0000 7fc9deffd640 1 -- 172.21.15.148:0/4054374812 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc9e007e8a0 con 0x7fc9e8141940 2023-10-30T12:18:07.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.236+0000 7fc9deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:18:07.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.236+0000 7fc9deffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:07.238 INFO:teuthology.orchestra.run.smithi148.stdout:64424509446 2023-10-30T12:18:07.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 10 librados: watch_flush enter 2023-10-30T12:18:07.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 10 librados: watch_flush exit 2023-10-30T12:18:07.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 20 client.4356.objecter shutdown clearing up homeless session... 2023-10-30T12:18:07.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 10 client.4356.objecter successfully canceled tick 2023-10-30T12:18:07.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/4054374812 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc9cc06c7c0 msgr2=0x7fc9cc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 --2- 172.21.15.148:0/4054374812 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc9cc06c7c0 0x7fc9cc06ec80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc9d80098f0 tx=0x7fc9d8008040 comp rx=0 tx=0).stop 2023-10-30T12:18:07.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 10 monclient: shutdown 2023-10-30T12:18:07.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:07.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/4054374812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e8141940 msgr2=0x7fc9e8139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 --2- 172.21.15.148:0/4054374812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e8141940 0x7fc9e8139bf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0036040 tx=0x7fc9e0004a40 comp rx=0 tx=0).stop 2023-10-30T12:18:07.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/4054374812 shutdown_connections 2023-10-30T12:18:07.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 --2- 172.21.15.148:0/4054374812 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc9cc06c7c0 0x7fc9cc06ec80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 --2- 172.21.15.148:0/4054374812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e8141940 0x7fc9e8139bf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 --2- 172.21.15.148:0/4054374812 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc9e80b7c10 0x7fc9e811f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 --2- 172.21.15.148:0/4054374812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e80b1c10 0x7fc9e81d6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/4054374812 >> 172.21.15.148:0/4054374812 conn(0x7fc9e806c660 msgr2=0x7fc9e813dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:07.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/4054374812 shutdown_connections 2023-10-30T12:18:07.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 -- 172.21.15.148:0/4054374812 wait complete. 2023-10-30T12:18:07.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.239+0000 7fc9ecdbb640 1 librados: shutdown 2023-10-30T12:18:07.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.240+0000 7fe2d97fa640 1 -- 172.21.15.148:0/2463144324 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe2c805e0d0 con 0x7fe2dc13e730 2023-10-30T12:18:07.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.240+0000 7fe2d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:18:07.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.240+0000 7fe2d97fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:07.244 INFO:teuthology.orchestra.run.smithi148.stdout:55834574854 2023-10-30T12:18:07.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 10 librados: watch_flush enter 2023-10-30T12:18:07.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 10 librados: watch_flush exit 2023-10-30T12:18:07.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 20 client.4316.objecter shutdown clearing up homeless session... 2023-10-30T12:18:07.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 10 client.4316.objecter successfully canceled tick 2023-10-30T12:18:07.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/2463144324 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe2c006c7f0 msgr2=0x7fe2c006ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 --2- 172.21.15.148:0/2463144324 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe2c006c7f0 0x7fe2c006ecb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009870 tx=0x7fe2d0030000 comp rx=0 tx=0).stop 2023-10-30T12:18:07.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 10 monclient: shutdown 2023-10-30T12:18:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/2463144324 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc13e730 msgr2=0x7fe2dc13a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 --2- 172.21.15.148:0/2463144324 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc13e730 0x7fe2dc13a750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800b410 tx=0x7fe2c800b8d0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/2463144324 shutdown_connections 2023-10-30T12:18:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 --2- 172.21.15.148:0/2463144324 >> v2:172.21.15.148:6800/3205090179 conn(0x7fe2c006c7f0 0x7fe2c006ecb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 --2- 172.21.15.148:0/2463144324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2dc141940 0x7fe2dc1da4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 --2- 172.21.15.148:0/2463144324 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe2dc13e730 0x7fe2dc13a750 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 --2- 172.21.15.148:0/2463144324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2dc0b7c10 0x7fe2dc1dadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/2463144324 >> 172.21.15.148:0/2463144324 conn(0x7fe2dc06c660 msgr2=0x7fe2dc141d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:07.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/2463144324 shutdown_connections 2023-10-30T12:18:07.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 -- 172.21.15.148:0/2463144324 wait complete. 2023-10-30T12:18:07.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.243+0000 7fe2e2e0e640 1 librados: shutdown 2023-10-30T12:18:07.252 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.1 2023-10-30T12:18:07.252 DEBUG:teuthology.parallel:result is None 2023-10-30T12:18:07.255 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2023-10-30T12:18:07.256 DEBUG:teuthology.parallel:result is None 2023-10-30T12:18:07.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.255+0000 7fd2b27fc640 1 -- 172.21.15.148:0/69133774 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fd2b403e450 con 0x7fd2bc141940 2023-10-30T12:18:07.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.255+0000 7fd2b27fc640 1 --2- 172.21.15.148:0/69133774 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd2a006c840 0x7fd2a006ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.255+0000 7fd2c1dfd640 1 --2- 172.21.15.148:0/69133774 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd2a006c840 0x7fd2a006ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.255+0000 7fd2c1dfd640 10 monclient: get_auth_request con 0x7fd2a006c840 auth_method 0 2023-10-30T12:18:07.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.256+0000 7fd2b27fc640 10 client.4396.objecter ms_handle_connect 0x7fd2a006c840 2023-10-30T12:18:07.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.256+0000 7fd2c1dfd640 1 --2- 172.21.15.148:0/69133774 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd2a006c840 0x7fd2a006ed00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009670 tx=0x7fd2b8006bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.257+0000 7fd2b27fc640 1 -- 172.21.15.148:0/69133774 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fd2b407b570 con 0x7fd2bc141940 2023-10-30T12:18:07.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.257+0000 7fd2b27fc640 10 client.4396.objecter ms_dispatch 0x7fd2bc000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:07.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.257+0000 7fd2b27fc640 3 client.4396.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:07.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.257+0000 7fd2b27fc640 3 client.4396.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:07.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.257+0000 7fd2b27fc640 20 client.4396.objecter dump_active .. 0 homeless 2023-10-30T12:18:07.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.266+0000 7fd2b27fc640 1 -- 172.21.15.148:0/69133774 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd2b407ae70 con 0x7fd2bc141940 2023-10-30T12:18:07.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.266+0000 7fd2b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.266+0000 7fd2b27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:07.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.273+0000 7f12f2ffd640 1 -- 172.21.15.148:0/3102636917 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f12f8060940 con 0x7f12fc0b1c10 2023-10-30T12:18:07.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.273+0000 7f12f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:18:07.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.273+0000 7f12f2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:07.274 INFO:teuthology.orchestra.run.smithi148.stdout:64424509446 2023-10-30T12:18:07.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 10 librados: watch_flush enter 2023-10-30T12:18:07.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 10 librados: watch_flush exit 2023-10-30T12:18:07.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 20 client.4393.objecter shutdown clearing up homeless session... 2023-10-30T12:18:07.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 10 client.4393.objecter successfully canceled tick 2023-10-30T12:18:07.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 -- 172.21.15.148:0/3102636917 >> v2:172.21.15.148:6800/3205090179 conn(0x7f12e006c810 msgr2=0x7f12e006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 --2- 172.21.15.148:0/3102636917 >> v2:172.21.15.148:6800/3205090179 conn(0x7f12e006c810 0x7f12e006ecd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0b1080 tx=0x7f12ec008040 comp rx=0 tx=0).stop 2023-10-30T12:18:07.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 10 monclient: shutdown 2023-10-30T12:18:07.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:07.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 -- 172.21.15.148:0/3102636917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 msgr2=0x7f12fc06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 --2- 172.21.15.148:0/3102636917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 0x7f12fc06d9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12f800e8b0 tx=0x7f12f800ed70 comp rx=0 tx=0).stop 2023-10-30T12:18:07.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 -- 172.21.15.148:0/3102636917 shutdown_connections 2023-10-30T12:18:07.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 --2- 172.21.15.148:0/3102636917 >> v2:172.21.15.148:6800/3205090179 conn(0x7f12e006c810 0x7f12e006ecd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 --2- 172.21.15.148:0/3102636917 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12fc141940 0x7f12fc1db470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 --2- 172.21.15.148:0/3102636917 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f12fc0b7c10 0x7f12fc0b0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 --2- 172.21.15.148:0/3102636917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12fc0b1c10 0x7f12fc06d9f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 -- 172.21.15.148:0/3102636917 >> 172.21.15.148:0/3102636917 conn(0x7f12fc06c660 msgr2=0x7f12fc13dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:07.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 -- 172.21.15.148:0/3102636917 shutdown_connections 2023-10-30T12:18:07.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 -- 172.21.15.148:0/3102636917 wait complete. 2023-10-30T12:18:07.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.275+0000 7f12f0ff9640 1 librados: shutdown 2023-10-30T12:18:07.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.7 2023-10-30T12:18:07.287 DEBUG:teuthology.parallel:result is None 2023-10-30T12:18:07.426 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:18:07.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.446+0000 7fd2c4061640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:18:07.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.446+0000 7fd2c4061640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:18:07.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.446+0000 7fd2c4061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.446+0000 7fd2c4061640 1 -- 172.21.15.148:0/69133774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd2bc0aa6a0 con 0x7fd2bc141940 2023-10-30T12:18:07.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.450+0000 7fd2b27fc640 1 -- 172.21.15.148:0/69133774 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd2b40810c0 con 0x7fd2bc141940 2023-10-30T12:18:07.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.450+0000 7fd2b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:18:07.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.450+0000 7fd2b27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:07.451 INFO:teuthology.orchestra.run.smithi148.stdout:64424509445 2023-10-30T12:18:07.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 10 librados: watch_flush enter 2023-10-30T12:18:07.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 10 librados: watch_flush exit 2023-10-30T12:18:07.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 20 client.4396.objecter shutdown clearing up homeless session... 2023-10-30T12:18:07.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 10 client.4396.objecter successfully canceled tick 2023-10-30T12:18:07.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 -- 172.21.15.148:0/69133774 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd2a006c840 msgr2=0x7fd2a006ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 --2- 172.21.15.148:0/69133774 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd2a006c840 0x7fd2a006ed00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009670 tx=0x7fd2b8006bf0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 10 monclient: shutdown 2023-10-30T12:18:07.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:07.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 -- 172.21.15.148:0/69133774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2bc141940 msgr2=0x7fd2bc1382a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 --2- 172.21.15.148:0/69133774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2bc141940 0x7fd2bc1382a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40093e0 tx=0x7fd2b4004a40 comp rx=0 tx=0).stop 2023-10-30T12:18:07.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 -- 172.21.15.148:0/69133774 shutdown_connections 2023-10-30T12:18:07.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 --2- 172.21.15.148:0/69133774 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd2a006c840 0x7fd2a006ed00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 --2- 172.21.15.148:0/69133774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2bc141940 0x7fd2bc1382a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 --2- 172.21.15.148:0/69133774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2bc13e730 0x7fd2bc137d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 --2- 172.21.15.148:0/69133774 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd2bc0b7c10 0x7fd2bc1da4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 -- 172.21.15.148:0/69133774 >> 172.21.15.148:0/69133774 conn(0x7fd2bc06c660 msgr2=0x7fd2bc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:07.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 -- 172.21.15.148:0/69133774 shutdown_connections 2023-10-30T12:18:07.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 -- 172.21.15.148:0/69133774 wait complete. 2023-10-30T12:18:07.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.452+0000 7fd2c4061640 1 librados: shutdown 2023-10-30T12:18:07.464 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.3 2023-10-30T12:18:07.464 DEBUG:teuthology.parallel:result is None 2023-10-30T12:18:07.519 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:18:07.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.532+0000 7fd7a14af640 10 monclient: get_monmap_and_config 2023-10-30T12:18:07.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.532+0000 7fd7a14af640 10 monclient: build_initial_monmap 2023-10-30T12:18:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.533+0000 7fd7a14af640 10 monclient: monmap: 2023-10-30T12:18:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:07.533025+0000 2023-10-30T12:18:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:07.533025+0000 2023-10-30T12:18:07.535 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:07.535 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.535 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:07.535 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:07.535 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:07.536 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:07.536 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:07.536 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.533+0000 7fd7a14af640 1 Processor -- start 2023-10-30T12:18:07.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.533+0000 7fd7a14af640 1 -- start start 2023-10-30T12:18:07.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.533+0000 7fd7a14af640 10 monclient: init 2023-10-30T12:18:07.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:07.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-10-30T12:18:07.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c0a9210 0x7fd79c071340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 10 monclient(hunting): picked mon.noname-b con 0x7fd79c0a9210 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:07.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd79c13b1d0 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:07.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd79c0af3f0 addr v1:172.21.15.148:6789/0 2023-10-30T12:18:07.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c0aa630 con 0x7fd79c0af3f0 2023-10-30T12:18:07.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c071de0 con 0x7fd79c13b1d0 2023-10-30T12:18:07.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c067340 con 0x7fd79c0a9210 2023-10-30T12:18:07.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:07.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.534+0000 7fd7a14af640 10 monclient(hunting): authenticate will time out at 1579.738115s 2023-10-30T12:18:07.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.563+0000 7fd79a7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd79c13b1d0 0x7fd79c0aece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:48334/0 (socket says 172.21.15.148:48334) 2023-10-30T12:18:07.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.563+0000 7fd79a7fc640 1 -- 172.21.15.148:0/3670025733 learned_addr learned my addr 172.21.15.148:0/3670025733 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:18:07.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.567+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391140820 0 0) 0x7fd794001490 con 0x7fd79c0af3f0 2023-10-30T12:18:07.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.567+0000 7fd799ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:18:07.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.567+0000 7fd799ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.567+0000 7fd799ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.567+0000 7fd799ffb640 10 monclient(hunting): my global_id is 4359 2023-10-30T12:18:07.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.567+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd784003520 con 0x7fd79c0af3f0 2023-10-30T12:18:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.568+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091549360 0 0) 0x7fd790001110 con 0x7fd79c13b1d0 2023-10-30T12:18:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.568+0000 7fd799ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:18:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.568+0000 7fd799ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.568+0000 7fd799ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.568+0000 7fd799ffb640 10 monclient(hunting): my global_id is 4399 2023-10-30T12:18:07.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.568+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd784003840 con 0x7fd79c13b1d0 2023-10-30T12:18:07.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.570+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2172769316 0 0) 0x7fd794001890 con 0x7fd79c0af3f0 2023-10-30T12:18:07.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.570+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c0a9210 msgr2=0x7fd79c071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:07.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.570+0000 7fd799ffb640 1 --2- 172.21.15.148:0/3670025733 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c0a9210 0x7fd79c071340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.570+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 >> v1:172.21.15.150:6789/0 conn(0x7fd79c13b1d0 legacy=0x7fd79c0aece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.570+0000 7fd799ffb640 10 monclient: _finish_hunting 0 2023-10-30T12:18:07.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.570+0000 7fd799ffb640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:18:07.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.570+0000 7fd799ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:07.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.570+0000 7fd799ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.148:6789/0 2023-10-30T12:18:07.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.570+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c0ac020 con 0x7fd79c0af3f0 2023-10-30T12:18:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient: _finish_auth 0 2023-10-30T12:18:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1567732308 0 0) 0x7fd794001d30 con 0x7fd79c0af3f0 2023-10-30T12:18:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd7a14af640 5 monclient: authenticate success, global_id 4359 2023-10-30T12:18:07.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd7a14af640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:07.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:18:07.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient: dump: 2023-10-30T12:18:07.575 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.577 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.577 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.577 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] but i'm connected to v1:172.21.15.148:6789/0, reconnecting 2023-10-30T12:18:07.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 >> v1:172.21.15.148:6789/0 conn(0x7fd79c0af3f0 legacy=0x7fd79c148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 1 --2- 172.21.15.148:0/3670025733 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c0a9210 0x7fd784003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient(hunting): picked mon.c con 0x7fd79c0a9210 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 1 --2- 172.21.15.148:0/3670025733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd784004270 0x7fd7840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient(hunting): picked mon.a con 0x7fd784004270 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:07.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 1 --2- 172.21.15.148:0/3670025733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd784004f50 0x7fd78400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient(hunting): picked mon.b con 0x7fd784004f50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:07.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c067340 con 0x7fd784004270 2023-10-30T12:18:07.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78400bec0 con 0x7fd784004f50 2023-10-30T12:18:07.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78400c010 con 0x7fd79c0a9210 2023-10-30T12:18:07.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:07.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd799ffb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:18:07.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.571+0000 7fd7a14af640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:07.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.572+0000 7fd79b7fe640 1 --2- 172.21.15.148:0/3670025733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd784004f50 0x7fd78400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.573+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request con 0x7fd784004f50 auth_method 0 2023-10-30T12:18:07.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.573+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.573+0000 7fd79b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.573+0000 7fd79b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:07.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.574+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:07.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.574+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:07.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.574+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:07.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.574+0000 7fd79affd640 1 --2- 172.21.15.148:0/3670025733 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c0a9210 0x7fd784003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79affd640 10 monclient(hunting): get_auth_request con 0x7fd79c0a9210 auth_method 0 2023-10-30T12:18:07.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:07.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_done global_id 4359 payload 293 2023-10-30T12:18:07.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 1 -- 172.21.15.148:0/3670025733 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c0a9210 msgr2=0x7fd784003cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:07.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 1 --2- 172.21.15.148:0/3670025733 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c0a9210 0x7fd784003cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 1 -- 172.21.15.148:0/3670025733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd784004270 msgr2=0x7fd7840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:07.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 1 --2- 172.21.15.148:0/3670025733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd784004270 0x7fd7840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:18:07.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 1 monclient: found mon.b 2023-10-30T12:18:07.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:07.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 1 -- 172.21.15.148:0/3670025733 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd794001890 con 0x7fd784004f50 2023-10-30T12:18:07.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:18:07.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:18:07.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.575+0000 7fd79b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:07.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.577+0000 7fd79b7fe640 1 --2- 172.21.15.148:0/3670025733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd784004f50 0x7fd78400b980 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0af800 tx=0x7fd7840057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4e53e7a79407bd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79402cda0 con 0x7fd784004f50 2023-10-30T12:18:07.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd799ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:07.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd799ffb640 10 monclient: dump: 2023-10-30T12:18:07.591 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.591 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.591 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.591 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.592 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.592 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.592 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.592 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.592 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.593 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd794006b50 con 0x7fd784004f50 2023-10-30T12:18:07.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:07.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd799ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:07.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:07.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 4 set_mon_vals no callback set 2023-10-30T12:18:07.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd799ffb640 1 -- 172.21.15.148:0/3670025733 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794007050 con 0x7fd784004f50 2023-10-30T12:18:07.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd799ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:07.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd799ffb640 10 monclient: dump: 2023-10-30T12:18:07.595 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.595 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.595 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.595 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.595 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.595 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.596 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.596 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.596 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.596 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 10 monclient: shutdown 2023-10-30T12:18:07.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:07.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 1 -- 172.21.15.148:0/3670025733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd784004f50 msgr2=0x7fd78400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 1 --2- 172.21.15.148:0/3670025733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd784004f50 0x7fd78400b980 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0af800 tx=0x7fd7840057f0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd79b7fe640 1 -- 172.21.15.148:0/3670025733 reap_dead start 2023-10-30T12:18:07.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 1 -- 172.21.15.148:0/3670025733 shutdown_connections 2023-10-30T12:18:07.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 1 -- 172.21.15.148:0/3670025733 >> 172.21.15.148:0/3670025733 conn(0x7fd79c0b3260 msgr2=0x7fd79c0b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:07.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 1 -- 172.21.15.148:0/3670025733 shutdown_connections 2023-10-30T12:18:07.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.581+0000 7fd7a14af640 1 -- 172.21.15.148:0/3670025733 wait complete. 2023-10-30T12:18:07.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.582+0000 7fd7a14af640 10 monclient: build_initial_monmap 2023-10-30T12:18:07.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.582+0000 7fd7a14af640 10 monclient: monmap: 2023-10-30T12:18:07.598 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:07.599 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:07.599 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:07.599 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:07.599 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:07.599 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:07.600 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:07.600 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:07.600 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:07.600 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.582+0000 7fd7a14af640 1 librados: starting msgr at 2023-10-30T12:18:07.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.582+0000 7fd7a14af640 1 librados: starting objecter 2023-10-30T12:18:07.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 1 Processor -- start 2023-10-30T12:18:07.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 1 -- start start 2023-10-30T12:18:07.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 1 librados: setting wanted keys 2023-10-30T12:18:07.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 1 librados: calling monclient init 2023-10-30T12:18:07.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient: init 2023-10-30T12:18:07.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:07.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:18:07.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd79c12e370 0x7fd79c13bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient(hunting): picked mon.noname-a con 0x7fd79c12e370 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:07.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c12e750 0x7fd79c13c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient(hunting): picked mon.noname-b con 0x7fd79c12e750 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:07.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c14ebb0 0x7fd79c14cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient(hunting): picked mon.noname-c con 0x7fd79c14ebb0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:07.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c0b3670 con 0x7fd79c12e370 2023-10-30T12:18:07.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c12cda0 con 0x7fd79c12e750 2023-10-30T12:18:07.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c12ec70 con 0x7fd79c14ebb0 2023-10-30T12:18:07.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:07.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.583+0000 7fd7a14af640 10 monclient(hunting): authenticate will time out at 1579.787603s 2023-10-30T12:18:07.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd79c12e370 0x7fd79c13bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79b7fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c14ebb0 0x7fd79c14cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79b7fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c14ebb0 0x7fd79c14cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54706/0 (socket says 172.21.15.148:54706) 2023-10-30T12:18:07.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79b7fe640 1 -- 172.21.15.148:0/2535559766 learned_addr learned my addr 172.21.15.148:0/2535559766 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:07.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request con 0x7fd79c14ebb0 auth_method 0 2023-10-30T12:18:07.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79a7fc640 1 --2- 172.21.15.148:0/2535559766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c12e750 0x7fd79c13c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79affd640 10 monclient(hunting): get_auth_request con 0x7fd79c12e370 auth_method 0 2023-10-30T12:18:07.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79a7fc640 10 monclient(hunting): get_auth_request con 0x7fd79c12e750 auth_method 0 2023-10-30T12:18:07.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.584+0000 7fd79a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.585+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:07.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.585+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:07.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.585+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:07.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.585+0000 7fd79a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:07.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.585+0000 7fd79a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:07.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.585+0000 7fd79a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:07.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.585+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:07.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.585+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:07.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.585+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:07.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.587+0000 7fd79a7fc640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2023-10-30T12:18:07.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.587+0000 7fd79a7fc640 1 -- 172.21.15.148:0/2535559766 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c14ebb0 msgr2=0x7fd79c14cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.587+0000 7fd79a7fc640 1 --2- 172.21.15.148:0/2535559766 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c14ebb0 0x7fd79c14cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.587+0000 7fd79a7fc640 1 -- 172.21.15.148:0/2535559766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd79c12e370 msgr2=0x7fd79c13bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.587+0000 7fd79a7fc640 1 --2- 172.21.15.148:0/2535559766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd79c12e370 0x7fd79c13bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.587+0000 7fd79a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:18:07.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.587+0000 7fd79a7fc640 1 monclient: found mon.noname-b 2023-10-30T12:18:07.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.587+0000 7fd79a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:07.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.587+0000 7fd79a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.587+0000 7fd79a7fc640 1 -- 172.21.15.148:0/2535559766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c14e3b0 con 0x7fd79c12e750 2023-10-30T12:18:07.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.589+0000 7fd79a7fc640 1 --2- 172.21.15.148:0/2535559766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c12e750 0x7fd79c13c1d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd79000c700 tx=0x7fd79000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.589+0000 7fd7837fe640 10 client.?.objecter ms_handle_connect 0x7fd79c12e750 2023-10-30T12:18:07.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.589+0000 7fd7837fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:07.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 1 -- 172.21.15.148:0/2535559766 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790004aa0 con 0x7fd79c12e750 2023-10-30T12:18:07.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:07.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 10 monclient: dump: 2023-10-30T12:18:07.615 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.615 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.615 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.615 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.616 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.616 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.616 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.616 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.616 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.617 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 10 monclient: _finish_auth 0 2023-10-30T12:18:07.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 10 monclient: _check_auth_tickets 2023-10-30T12:18:07.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:07.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 1 -- 172.21.15.148:0/2535559766 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79000e040 con 0x7fd79c12e750 2023-10-30T12:18:07.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7a14af640 5 monclient: authenticate success, global_id 4402 2023-10-30T12:18:07.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7a14af640 10 monclient: _renew_subs 2023-10-30T12:18:07.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7a14af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7a14af640 1 -- 172.21.15.148:0/2535559766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79c13dad0 con 0x7fd79c12e750 2023-10-30T12:18:07.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:07.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 1 -- 172.21.15.148:0/2535559766 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790007070 con 0x7fd79c12e750 2023-10-30T12:18:07.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:07.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7837fe640 10 monclient: dump: 2023-10-30T12:18:07.619 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.619 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.620 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.620 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.620 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.620 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.620 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.621 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7997fa640 4 set_mon_vals no callback set 2023-10-30T12:18:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7a14af640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7a14af640 10 monclient: _renew_subs 2023-10-30T12:18:07.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7a14af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7a14af640 1 -- 172.21.15.148:0/2535559766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79c14d600 con 0x7fd79c12e750 2023-10-30T12:18:07.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.593+0000 7fd7a14af640 1 librados: init done 2023-10-30T12:18:07.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.594+0000 7fd7a14af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.594+0000 7fd7a14af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.594+0000 7fd7a14af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.594+0000 7fd7a14af640 1 -- 172.21.15.148:0/2535559766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79c14e5b0 con 0x7fd79c12e750 2023-10-30T12:18:07.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.611+0000 7fd7837fe640 1 -- 172.21.15.148:0/2535559766 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fd7900071e0 con 0x7fd79c12e750 2023-10-30T12:18:07.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.611+0000 7fd7837fe640 1 --2- 172.21.15.148:0/2535559766 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd78806c7c0 0x7fd78806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.611+0000 7fd79affd640 1 --2- 172.21.15.148:0/2535559766 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd78806c7c0 0x7fd78806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.611+0000 7fd79affd640 10 monclient: get_auth_request con 0x7fd78806c7c0 auth_method 0 2023-10-30T12:18:07.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.611+0000 7fd7837fe640 10 client.4402.objecter ms_handle_connect 0x7fd78806c7c0 2023-10-30T12:18:07.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.611+0000 7fd79affd640 1 --2- 172.21.15.148:0/2535559766 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd78806c7c0 0x7fd78806ec80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd78c004700 tx=0x7fd78c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.624 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:18:07.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.613+0000 7fd7837fe640 1 -- 172.21.15.148:0/2535559766 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fd790022080 con 0x7fd79c12e750 2023-10-30T12:18:07.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.613+0000 7fd7837fe640 10 client.4402.objecter ms_dispatch 0x7fd79c000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:07.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.613+0000 7fd7837fe640 3 client.4402.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:07.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.613+0000 7fd7837fe640 3 client.4402.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:07.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.613+0000 7fd7837fe640 20 client.4402.objecter dump_active .. 0 homeless 2023-10-30T12:18:07.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.628+0000 7facd3573640 10 monclient: get_monmap_and_config 2023-10-30T12:18:07.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.628+0000 7facd3573640 10 monclient: build_initial_monmap 2023-10-30T12:18:07.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.628+0000 7facd3573640 10 monclient: monmap: 2023-10-30T12:18:07.629 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:07.629 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.629 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:07.628297+0000 2023-10-30T12:18:07.630 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:07.628297+0000 2023-10-30T12:18:07.630 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:07.630 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.630 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:07.630 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:07.631 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:07.631 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:07.631 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:07.631 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.628+0000 7facd3573640 1 Processor -- start 2023-10-30T12:18:07.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 1 -- start start 2023-10-30T12:18:07.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 10 monclient: init 2023-10-30T12:18:07.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:07.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-10-30T12:18:07.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faccc0b7c10 0x7faccc0a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 10 monclient(hunting): picked mon.noname-c con 0x7faccc0b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:07.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7faccc141940 0x7faccc141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 10 monclient(hunting): picked mon.noname-a con 0x7faccc141940 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:07.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faccc0b1c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:07.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0a9350 con 0x7faccc0b1c10 2023-10-30T12:18:07.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faccc0670a0 con 0x7faccc141940 2023-10-30T12:18:07.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faccc0b7ff0 con 0x7faccc0b7c10 2023-10-30T12:18:07.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:07.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.629+0000 7facd3573640 10 monclient(hunting): authenticate will time out at 1579.833509s 2023-10-30T12:18:07.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.630+0000 7facd0b0e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7faccc141940 0x7faccc141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.630+0000 7facd0b0e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7faccc141940 0x7faccc141d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60164/0 (socket says 172.21.15.148:60164) 2023-10-30T12:18:07.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.630+0000 7facd0b0e640 1 -- 172.21.15.148:0/2704799422 learned_addr learned my addr 172.21.15.148:0/2704799422 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:07.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.630+0000 7facd0b0e640 10 monclient(hunting): get_auth_request con 0x7faccc141940 auth_method 0 2023-10-30T12:18:07.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.630+0000 7facd0b0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.630+0000 7facd0b0e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.630+0000 7facd0b0e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.630+0000 7facd0b0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:07.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.630+0000 7facd0b0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:07.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.631+0000 7facd0b0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:07.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.632+0000 7facd0b0e640 10 monclient(hunting): handle_auth_done global_id 4365 payload 274 2023-10-30T12:18:07.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.632+0000 7facd0b0e640 1 -- 172.21.15.148:0/2704799422 >> v2:172.21.15.150:3300/0 conn(0x7faccc0b7c10 msgr2=0x7faccc0a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:07.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.632+0000 7facd0b0e640 1 --2- 172.21.15.148:0/2704799422 >> v2:172.21.15.150:3300/0 conn(0x7faccc0b7c10 0x7faccc0a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.632+0000 7facd0b0e640 1 -- 172.21.15.148:0/2704799422 >> v1:172.21.15.150:6789/0 conn(0x7faccc0b1c10 legacy=0x7faccc0b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:07.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.632+0000 7facd0b0e640 10 monclient: _finish_hunting 0 2023-10-30T12:18:07.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.632+0000 7facd0b0e640 1 monclient: found mon.noname-a 2023-10-30T12:18:07.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.632+0000 7facd0b0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:07.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.632+0000 7facd0b0e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:07.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.632+0000 7facd0b0e640 1 -- 172.21.15.148:0/2704799422 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0aa6a0 con 0x7faccc141940 2023-10-30T12:18:07.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.634+0000 7facd0b0e640 1 --2- 172.21.15.148:0/2704799422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faccc141940 0x7faccc141d90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faccc0b4d80 tx=0x7facc802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f36d4c774bdf69d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 1 -- 172.21.15.148:0/2704799422 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc80317e0 con 0x7faccc141940 2023-10-30T12:18:07.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:07.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 10 monclient: dump: 2023-10-30T12:18:07.643 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.643 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.643 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.644 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.644 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.644 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.644 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.644 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.645 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.645 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 10 monclient: _finish_auth 0 2023-10-30T12:18:07.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:07.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:07.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 1 -- 172.21.15.148:0/2704799422 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc8031950 con 0x7faccc141940 2023-10-30T12:18:07.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facd3573640 5 monclient: authenticate success, global_id 4365 2023-10-30T12:18:07.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facd3573640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:07.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:07.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facd3573640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:07.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facd3573640 4 set_mon_vals no callback set 2023-10-30T12:18:07.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 1 -- 172.21.15.148:0/2704799422 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8031cf0 con 0x7faccc141940 2023-10-30T12:18:07.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:07.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.638+0000 7facc3fff640 10 monclient: dump: 2023-10-30T12:18:07.647 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.648 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.648 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.648 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.648 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.648 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.648 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.649 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.649 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.649 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.639+0000 7facd3573640 10 monclient: shutdown 2023-10-30T12:18:07.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.639+0000 7facd3573640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:07.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.639+0000 7facd3573640 1 -- 172.21.15.148:0/2704799422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faccc141940 msgr2=0x7faccc141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.639+0000 7facd3573640 1 --2- 172.21.15.148:0/2704799422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faccc141940 0x7faccc141d90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faccc0b4d80 tx=0x7facc802b880 comp rx=0 tx=0).stop 2023-10-30T12:18:07.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.639+0000 7facd3573640 1 -- 172.21.15.148:0/2704799422 shutdown_connections 2023-10-30T12:18:07.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.639+0000 7facd3573640 1 --2- 172.21.15.148:0/2704799422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faccc141940 0x7faccc141d90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.639+0000 7facd3573640 1 --2- 172.21.15.148:0/2704799422 >> v2:172.21.15.150:3300/0 conn(0x7faccc0b7c10 0x7faccc0a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.639+0000 7facd3573640 1 -- 172.21.15.148:0/2704799422 >> 172.21.15.148:0/2704799422 conn(0x7faccc06c660 msgr2=0x7faccc0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:07.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.639+0000 7facd3573640 1 -- 172.21.15.148:0/2704799422 shutdown_connections 2023-10-30T12:18:07.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.639+0000 7facd3573640 1 -- 172.21.15.148:0/2704799422 wait complete. 2023-10-30T12:18:07.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.640+0000 7facd3573640 10 monclient: build_initial_monmap 2023-10-30T12:18:07.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.640+0000 7facd3573640 10 monclient: monmap: 2023-10-30T12:18:07.651 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:07.652 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:07.652 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:07.652 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:07.652 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:07.652 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:07.653 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:07.653 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:07.653 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:07.653 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.640+0000 7facd3573640 1 librados: starting msgr at 2023-10-30T12:18:07.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.640+0000 7facd3573640 1 librados: starting objecter 2023-10-30T12:18:07.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.640+0000 7facd3573640 1 Processor -- start 2023-10-30T12:18:07.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.640+0000 7facd3573640 1 -- start start 2023-10-30T12:18:07.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.640+0000 7facd3573640 1 librados: setting wanted keys 2023-10-30T12:18:07.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.640+0000 7facd3573640 1 librados: calling monclient init 2023-10-30T12:18:07.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.640+0000 7facd3573640 10 monclient: init 2023-10-30T12:18:07.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:07.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:07.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faccc0b1c10 0x7faccc1da0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 10 monclient(hunting): picked mon.noname-a con 0x7faccc0b1c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:07.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7faccc0b7c10 0x7faccc1da930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 10 monclient(hunting): picked mon.noname-c con 0x7faccc0b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:07.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc141940 0x7faccc0ac280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 10 monclient(hunting): picked mon.noname-b con 0x7faccc141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:07.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc0b2880 con 0x7faccc0b1c10 2023-10-30T12:18:07.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc1c7b50 con 0x7faccc141940 2023-10-30T12:18:07.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faccc0b02d0 con 0x7faccc0b7c10 2023-10-30T12:18:07.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:07.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd3573640 10 monclient(hunting): authenticate will time out at 1579.845229s 2023-10-30T12:18:07.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd130f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faccc0b1c10 0x7faccc1da0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.641+0000 7facd0b0e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7faccc0b7c10 0x7faccc1da930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd130f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faccc0b1c10 0x7faccc1da0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60180/0 (socket says 172.21.15.148:60180) 2023-10-30T12:18:07.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd130f640 1 -- 172.21.15.148:0/3189656847 learned_addr learned my addr 172.21.15.148:0/3189656847 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:07.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd130f640 10 monclient(hunting): get_auth_request con 0x7faccc0b1c10 auth_method 0 2023-10-30T12:18:07.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd130f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd130f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd130f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd0b0e640 10 monclient(hunting): get_auth_request con 0x7faccc0b7c10 auth_method 0 2023-10-30T12:18:07.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd0b0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd0b0e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd0b0e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd130f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:07.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd130f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:07.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.642+0000 7facd130f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:07.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.643+0000 7facd0b0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:07.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.643+0000 7facd0b0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:07.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.643+0000 7facd0b0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:07.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.647+0000 7facd0b0e640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2023-10-30T12:18:07.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.647+0000 7facd0b0e640 1 -- 172.21.15.148:0/3189656847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc141940 msgr2=0x7faccc0ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.647+0000 7facd0b0e640 1 --2- 172.21.15.148:0/3189656847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc141940 0x7faccc0ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.647+0000 7facd0b0e640 1 -- 172.21.15.148:0/3189656847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faccc0b1c10 msgr2=0x7faccc1da0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.647+0000 7facd0b0e640 1 --2- 172.21.15.148:0/3189656847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faccc0b1c10 0x7faccc1da0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.647+0000 7facd0b0e640 10 monclient: _finish_hunting 0 2023-10-30T12:18:07.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.647+0000 7facd0b0e640 1 monclient: found mon.noname-c 2023-10-30T12:18:07.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.647+0000 7facd0b0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:07.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.647+0000 7facd0b0e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:18:07.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.647+0000 7facd0b0e640 1 -- 172.21.15.148:0/3189656847 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0ae0f0 con 0x7faccc0b7c10 2023-10-30T12:18:07.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.649+0000 7facd0b0e640 1 --2- 172.21.15.148:0/3189656847 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7faccc0b7c10 0x7faccc1da930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faccc0b2d10 tx=0x7facc80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.649+0000 7facc1ffb640 10 client.?.objecter ms_handle_connect 0x7faccc0b7c10 2023-10-30T12:18:07.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.649+0000 7facc1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:07.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.653+0000 7facc1ffb640 1 -- 172.21.15.148:0/3189656847 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc80329c0 con 0x7faccc0b7c10 2023-10-30T12:18:07.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.653+0000 7facc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.653+0000 7facc1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:07.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.653+0000 7facc1ffb640 10 monclient: dump: 2023-10-30T12:18:07.666 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.666 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.667 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.667 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.667 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.667 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.667 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.668 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.668 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.668 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facc1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:18:07.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facc1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:07.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facc1ffb640 1 -- 172.21.15.148:0/3189656847 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc803f030 con 0x7faccc0b7c10 2023-10-30T12:18:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facd3573640 5 monclient: authenticate success, global_id 4322 2023-10-30T12:18:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facd3573640 10 monclient: _renew_subs 2023-10-30T12:18:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facd3573640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facd3573640 1 -- 172.21.15.148:0/3189656847 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc0ac7c0 con 0x7faccc0b7c10 2023-10-30T12:18:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facc1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facc1ffb640 1 -- 172.21.15.148:0/3189656847 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8032db0 con 0x7faccc0b7c10 2023-10-30T12:18:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facc1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facc1ffb640 10 monclient: dump: 2023-10-30T12:18:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.672 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.672 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.672 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.672 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facc37fe640 4 set_mon_vals no callback set 2023-10-30T12:18:07.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facd3573640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facd3573640 10 monclient: _renew_subs 2023-10-30T12:18:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facd3573640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facd3573640 1 -- 172.21.15.148:0/3189656847 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc137fd0 con 0x7faccc0b7c10 2023-10-30T12:18:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.654+0000 7facd3573640 1 librados: init done 2023-10-30T12:18:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.655+0000 7facd3573640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.655+0000 7facd3573640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.655+0000 7facd3573640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:07.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.655+0000 7facd3573640 1 -- 172.21.15.148:0/3189656847 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc0ac960 con 0x7faccc0b7c10 2023-10-30T12:18:07.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.728+0000 7f3260989640 10 monclient: get_monmap_and_config 2023-10-30T12:18:07.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.728+0000 7f3260989640 10 monclient: build_initial_monmap 2023-10-30T12:18:07.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.728+0000 7f3260989640 10 monclient: monmap: 2023-10-30T12:18:07.729 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:07.730 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.730 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:07.728098+0000 2023-10-30T12:18:07.730 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:07.728098+0000 2023-10-30T12:18:07.730 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:07.731 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.731 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:07.731 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:07.731 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:07.731 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:07.731 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:07.732 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.728+0000 7f3260989640 1 Processor -- start 2023-10-30T12:18:07.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.728+0000 7f3260989640 1 -- start start 2023-10-30T12:18:07.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.728+0000 7f3260989640 10 monclient: init 2023-10-30T12:18:07.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:07.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-10-30T12:18:07.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f325c0b7c10 0x7f325c0a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 10 monclient(hunting): picked mon.noname-c con 0x7f325c0b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:07.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f325c141940 0x7f325c141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 10 monclient(hunting): picked mon.noname-b con 0x7f325c141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:07.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f325c0b1c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:07.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c0a9350 con 0x7f325c0b1c10 2023-10-30T12:18:07.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c0670a0 con 0x7f325c0b7c10 2023-10-30T12:18:07.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0b7ff0 con 0x7f325c141940 2023-10-30T12:18:07.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:07.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.729+0000 7f3260989640 10 monclient(hunting): authenticate will time out at 1579.933370s 2023-10-30T12:18:07.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.849+0000 7fd7837fe640 1 -- 172.21.15.148:0/2535559766 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd7900576a0 con 0x7fd79c12e750 2023-10-30T12:18:07.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.849+0000 7fd7837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.849+0000 7fd7837fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:07.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.856+0000 7f325a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f325c0b7c10 0x7f325c0a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.857+0000 7f325a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f325c0b7c10 0x7f325c0a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:49072/0 (socket says 172.21.15.148:49072) 2023-10-30T12:18:07.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.857+0000 7f325a59c640 1 -- 172.21.15.148:0/1973304133 learned_addr learned my addr 172.21.15.148:0/1973304133 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:07.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.857+0000 7f325a59c640 10 monclient(hunting): get_auth_request con 0x7f325c0b7c10 auth_method 0 2023-10-30T12:18:07.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.857+0000 7f325a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.857+0000 7f325a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.857+0000 7f325a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.859+0000 7f325a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:07.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.859+0000 7f325a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:07.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.859+0000 7f325a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:07.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.862+0000 7f325a59c640 10 monclient(hunting): handle_auth_done global_id 4405 payload 274 2023-10-30T12:18:07.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.862+0000 7f325a59c640 1 -- 172.21.15.148:0/1973304133 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f325c141940 msgr2=0x7f325c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:07.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.862+0000 7f325a59c640 1 --2- 172.21.15.148:0/1973304133 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f325c141940 0x7f325c141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.862+0000 7f325a59c640 1 -- 172.21.15.148:0/1973304133 >> v1:172.21.15.150:6789/0 conn(0x7f325c0b1c10 legacy=0x7f325c0b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.862+0000 7f325a59c640 10 monclient: _finish_hunting 0 2023-10-30T12:18:07.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.862+0000 7f325a59c640 1 monclient: found mon.noname-c 2023-10-30T12:18:07.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.862+0000 7f325a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:07.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.862+0000 7f325a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.862+0000 7f325a59c640 1 -- 172.21.15.148:0/1973304133 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c0aa6a0 con 0x7f325c0b7c10 2023-10-30T12:18:07.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.865+0000 7f325a59c640 1 --2- 172.21.15.148:0/1973304133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f325c0b7c10 0x7f325c0a8aa0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f324c009870 tx=0x7f324c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=416a73a661f4a9ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 1 -- 172.21.15.148:0/1973304133 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c031640 con 0x7f325c0b7c10 2023-10-30T12:18:07.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:07.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 10 monclient: dump: 2023-10-30T12:18:07.870 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.871 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.871 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.871 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.871 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.871 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.872 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.872 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.872 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.872 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 10 monclient: _finish_auth 0 2023-10-30T12:18:07.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:18:07.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:07.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 1 -- 172.21.15.148:0/1973304133 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c0317b0 con 0x7f325c0b7c10 2023-10-30T12:18:07.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:07.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 1 -- 172.21.15.148:0/1973304133 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c031b50 con 0x7f325c0b7c10 2023-10-30T12:18:07.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3260989640 5 monclient: authenticate success, global_id 4405 2023-10-30T12:18:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3260989640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3260989640 4 set_mon_vals no callback set 2023-10-30T12:18:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.869+0000 7f3259d9b640 10 monclient: dump: 2023-10-30T12:18:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.876 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.870+0000 7f3260989640 10 monclient: shutdown 2023-10-30T12:18:07.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.870+0000 7f3260989640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:07.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.870+0000 7f3260989640 1 -- 172.21.15.148:0/1973304133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f325c0b7c10 msgr2=0x7f325c0a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.870+0000 7f3260989640 1 --2- 172.21.15.148:0/1973304133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f325c0b7c10 0x7f325c0a8aa0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f324c009870 tx=0x7f324c02b8f0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.870+0000 7f3260989640 1 -- 172.21.15.148:0/1973304133 shutdown_connections 2023-10-30T12:18:07.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.870+0000 7f3260989640 1 --2- 172.21.15.148:0/1973304133 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f325c141940 0x7f325c141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.870+0000 7f3260989640 1 --2- 172.21.15.148:0/1973304133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f325c0b7c10 0x7f325c0a8aa0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.870+0000 7f3260989640 1 -- 172.21.15.148:0/1973304133 >> 172.21.15.148:0/1973304133 conn(0x7f325c06c660 msgr2=0x7f325c0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:07.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.870+0000 7f3260989640 1 -- 172.21.15.148:0/1973304133 shutdown_connections 2023-10-30T12:18:07.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.870+0000 7f3260989640 1 -- 172.21.15.148:0/1973304133 wait complete. 2023-10-30T12:18:07.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 10 monclient: build_initial_monmap 2023-10-30T12:18:07.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 10 monclient: monmap: 2023-10-30T12:18:07.882 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:07.882 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:07.882 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:07.883 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:07.883 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:07.883 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:07.883 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:07.883 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:07.883 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:07.884 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 1 librados: starting msgr at 2023-10-30T12:18:07.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 1 librados: starting objecter 2023-10-30T12:18:07.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 1 Processor -- start 2023-10-30T12:18:07.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 1 -- start start 2023-10-30T12:18:07.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 1 librados: setting wanted keys 2023-10-30T12:18:07.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 1 librados: calling monclient init 2023-10-30T12:18:07.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 10 monclient: init 2023-10-30T12:18:07.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:07.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:18:07.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.871+0000 7f3260989640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f325c0b1c10 0x7f325c0ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 10 monclient(hunting): picked mon.noname-c con 0x7f325c0b1c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:07.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f325c0b7c10 0x7f325c1c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 10 monclient(hunting): picked mon.noname-b con 0x7f325c0b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:07.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f325c141940 0x7f325c1c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 10 monclient(hunting): picked mon.noname-a con 0x7f325c141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:07.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0a84d0 con 0x7f325c141940 2023-10-30T12:18:07.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c1c9c70 con 0x7f325c0b7c10 2023-10-30T12:18:07.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:07.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0b0290 con 0x7f325c0b1c10 2023-10-30T12:18:07.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:07.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3260989640 10 monclient(hunting): authenticate will time out at 1580.075951s 2023-10-30T12:18:07.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f325ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f325c141940 0x7f325c1c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f3253fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f325c0b7c10 0x7f325c1c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f325ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f325c141940 0x7f325c1c8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60184/0 (socket says 172.21.15.148:60184) 2023-10-30T12:18:07.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f325ad9d640 1 -- 172.21.15.148:0/1488302932 learned_addr learned my addr 172.21.15.148:0/1488302932 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:07.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f325ad9d640 10 monclient(hunting): get_auth_request con 0x7f325c141940 auth_method 0 2023-10-30T12:18:07.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f325ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f325ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.872+0000 7f325ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.873+0000 7f3253fff640 10 monclient(hunting): get_auth_request con 0x7f325c0b7c10 auth_method 0 2023-10-30T12:18:07.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.873+0000 7f3253fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:07.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.873+0000 7f3253fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:07.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.873+0000 7f3253fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.873+0000 7f325ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.873+0000 7f325ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.873+0000 7f325ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.873+0000 7f3253fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.873+0000 7f3253fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.873+0000 7f3253fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.876+0000 7f3253fff640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2023-10-30T12:18:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.876+0000 7f3253fff640 1 -- 172.21.15.148:0/1488302932 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f325c0b1c10 msgr2=0x7f325c0ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.876+0000 7f3253fff640 1 --2- 172.21.15.148:0/1488302932 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f325c0b1c10 0x7f325c0ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.876+0000 7f3253fff640 1 -- 172.21.15.148:0/1488302932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f325c141940 msgr2=0x7f325c1c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.876+0000 7f3253fff640 1 --2- 172.21.15.148:0/1488302932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f325c141940 0x7f325c1c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:07.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.876+0000 7f3253fff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:07.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.876+0000 7f3253fff640 1 monclient: found mon.noname-b 2023-10-30T12:18:07.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.876+0000 7f3253fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:07.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.876+0000 7f3253fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.876+0000 7f3253fff640 1 -- 172.21.15.148:0/1488302932 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c13b650 con 0x7f325c0b7c10 2023-10-30T12:18:07.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.878+0000 7f3253fff640 1 --2- 172.21.15.148:0/1488302932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f325c0b7c10 0x7f325c1c83e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f324400b410 tx=0x7f324400b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.878+0000 7f3252ffd640 10 client.?.objecter ms_handle_connect 0x7f325c0b7c10 2023-10-30T12:18:07.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.878+0000 7f3252ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 1 -- 172.21.15.148:0/1488302932 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244011830 con 0x7f325c0b7c10 2023-10-30T12:18:07.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:07.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 10 monclient: dump: 2023-10-30T12:18:07.896 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.896 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.897 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.897 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.897 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.897 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.897 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.898 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3260989640 5 monclient: authenticate success, global_id 4408 2023-10-30T12:18:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 1 -- 172.21.15.148:0/1488302932 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324400d040 con 0x7f325c0b7c10 2023-10-30T12:18:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 1 -- 172.21.15.148:0/1488302932 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244011c00 con 0x7f325c0b7c10 2023-10-30T12:18:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3260989640 10 monclient: _renew_subs 2023-10-30T12:18:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3260989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3260989640 1 -- 172.21.15.148:0/1488302932 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c0ae940 con 0x7f325c0b7c10 2023-10-30T12:18:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f325959a640 4 set_mon_vals no callback set 2023-10-30T12:18:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3252ffd640 10 monclient: dump: 2023-10-30T12:18:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:07.902 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3260989640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3260989640 10 monclient: _renew_subs 2023-10-30T12:18:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3260989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3260989640 1 -- 172.21.15.148:0/1488302932 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c1c7d30 con 0x7f325c0b7c10 2023-10-30T12:18:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.882+0000 7f3260989640 1 librados: init done 2023-10-30T12:18:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.883+0000 7f3260989640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.883+0000 7f3260989640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.883+0000 7f3260989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.883+0000 7f3260989640 1 -- 172.21.15.148:0/1488302932 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c0a97c0 con 0x7f325c0b7c10 2023-10-30T12:18:07.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.913+0000 7facc1ffb640 1 -- 172.21.15.148:0/3189656847 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7facc80453f0 con 0x7faccc0b7c10 2023-10-30T12:18:07.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.913+0000 7facc1ffb640 1 --2- 172.21.15.148:0/3189656847 >> v2:172.21.15.148:6800/3205090179 conn(0x7facb006c810 0x7facb006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:07.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.913+0000 7facd130f640 1 --2- 172.21.15.148:0/3189656847 >> v2:172.21.15.148:6800/3205090179 conn(0x7facb006c810 0x7facb006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:07.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.913+0000 7facd130f640 10 monclient: get_auth_request con 0x7facb006c810 auth_method 0 2023-10-30T12:18:07.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.914+0000 7facd130f640 1 --2- 172.21.15.148:0/3189656847 >> v2:172.21.15.148:6800/3205090179 conn(0x7facb006c810 0x7facb006ecd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7facbc009a40 tx=0x7facbc0091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:07.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.914+0000 7facc1ffb640 10 client.4322.objecter ms_handle_connect 0x7facb006c810 2023-10-30T12:18:07.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.926+0000 7facc1ffb640 1 -- 172.21.15.148:0/3189656847 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7facc807a4c0 con 0x7faccc0b7c10 2023-10-30T12:18:07.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.926+0000 7facc1ffb640 10 client.4322.objecter ms_dispatch 0x7faccc000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:07.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.926+0000 7facc1ffb640 3 client.4322.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:07.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.926+0000 7facc1ffb640 3 client.4322.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:07.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.926+0000 7facc1ffb640 20 client.4322.objecter dump_active .. 0 homeless 2023-10-30T12:18:07.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.926+0000 7facc1ffb640 1 -- 172.21.15.148:0/3189656847 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7facc8037070 con 0x7faccc0b7c10 2023-10-30T12:18:07.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.926+0000 7facc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:07.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:07.926+0000 7facc1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:08.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.038+0000 7fd7a14af640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:18:08.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.038+0000 7fd7a14af640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:18:08.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.038+0000 7fd7a14af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:08.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.038+0000 7fd7a14af640 1 -- 172.21.15.148:0/2535559766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd79c0aaaa0 con 0x7fd79c12e750 2023-10-30T12:18:08.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.103+0000 7facd3573640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:18:08.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.103+0000 7facd3573640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:18:08.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.103+0000 7facd3573640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:18:08.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.103+0000 7facd3573640 1 -- 172.21.15.148:0/3189656847 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7faccc1da650 con 0x7faccc0b7c10 2023-10-30T12:18:08.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.106+0000 7facc1ffb640 1 -- 172.21.15.148:0/3189656847 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7facc8079dc0 con 0x7faccc0b7c10 2023-10-30T12:18:08.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.106+0000 7facc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:18:08.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.106+0000 7facc1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:08.108 INFO:teuthology.orchestra.run.smithi148.stdout:64424509446 2023-10-30T12:18:08.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 10 librados: watch_flush enter 2023-10-30T12:18:08.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 10 librados: watch_flush exit 2023-10-30T12:18:08.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 20 client.4322.objecter shutdown clearing up homeless session... 2023-10-30T12:18:08.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 10 client.4322.objecter successfully canceled tick 2023-10-30T12:18:08.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 -- 172.21.15.148:0/3189656847 >> v2:172.21.15.148:6800/3205090179 conn(0x7facb006c810 msgr2=0x7facb006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 --2- 172.21.15.148:0/3189656847 >> v2:172.21.15.148:6800/3205090179 conn(0x7facb006c810 0x7facb006ecd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7facbc009a40 tx=0x7facbc0091c0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 10 monclient: shutdown 2023-10-30T12:18:08.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:08.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 -- 172.21.15.148:0/3189656847 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7faccc0b7c10 msgr2=0x7faccc1da930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 --2- 172.21.15.148:0/3189656847 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7faccc0b7c10 0x7faccc1da930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faccc0b2d10 tx=0x7facc80046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 -- 172.21.15.148:0/3189656847 shutdown_connections 2023-10-30T12:18:08.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 --2- 172.21.15.148:0/3189656847 >> v2:172.21.15.148:6800/3205090179 conn(0x7facb006c810 0x7facb006ecd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 --2- 172.21.15.148:0/3189656847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc141940 0x7faccc0ac280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 --2- 172.21.15.148:0/3189656847 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7faccc0b7c10 0x7faccc1da930 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 --2- 172.21.15.148:0/3189656847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faccc0b1c10 0x7faccc1da0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 -- 172.21.15.148:0/3189656847 >> 172.21.15.148:0/3189656847 conn(0x7faccc06c660 msgr2=0x7faccc0b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:08.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.108+0000 7facd3573640 1 -- 172.21.15.148:0/3189656847 shutdown_connections 2023-10-30T12:18:08.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.109+0000 7facd3573640 1 -- 172.21.15.148:0/3189656847 wait complete. 2023-10-30T12:18:08.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.109+0000 7facd3573640 1 librados: shutdown 2023-10-30T12:18:08.121 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.5 2023-10-30T12:18:08.121 DEBUG:teuthology.parallel:result is None 2023-10-30T12:18:08.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.126+0000 7f3252ffd640 1 -- 172.21.15.148:0/1488302932 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f324401a400 con 0x7f325c0b7c10 2023-10-30T12:18:08.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.126+0000 7f3252ffd640 1 --2- 172.21.15.148:0/1488302932 >> v2:172.21.15.148:6800/3205090179 conn(0x7f323c06c810 0x7f323c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:08.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.126+0000 7f325a59c640 1 --2- 172.21.15.148:0/1488302932 >> v2:172.21.15.148:6800/3205090179 conn(0x7f323c06c810 0x7f323c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:08.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.126+0000 7f325a59c640 10 monclient: get_auth_request con 0x7f323c06c810 auth_method 0 2023-10-30T12:18:08.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.127+0000 7f325a59c640 1 --2- 172.21.15.148:0/1488302932 >> v2:172.21.15.148:6800/3205090179 conn(0x7f323c06c810 0x7f323c06ecd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f324c036000 tx=0x7f324c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:08.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.127+0000 7f3252ffd640 10 client.4408.objecter ms_handle_connect 0x7f323c06c810 2023-10-30T12:18:08.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.129+0000 7f3252ffd640 1 -- 172.21.15.148:0/1488302932 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f3244058850 con 0x7f325c0b7c10 2023-10-30T12:18:08.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.129+0000 7f3252ffd640 10 client.4408.objecter ms_dispatch 0x7f325c000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:08.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.129+0000 7f3252ffd640 3 client.4408.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:08.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.129+0000 7f3252ffd640 3 client.4408.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:08.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.129+0000 7f3252ffd640 20 client.4408.objecter dump_active .. 0 homeless 2023-10-30T12:18:08.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.139+0000 7f3252ffd640 1 -- 172.21.15.148:0/1488302932 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3244058150 con 0x7f325c0b7c10 2023-10-30T12:18:08.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.139+0000 7f3252ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:08.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.139+0000 7f3252ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:08.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.187+0000 7fd7837fe640 1 -- 172.21.15.148:0/2535559766 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd79005b740 con 0x7fd79c12e750 2023-10-30T12:18:08.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.187+0000 7fd7837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:18:08.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.187+0000 7fd7837fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:08.189 INFO:teuthology.orchestra.run.smithi148.stdout:64424509446 2023-10-30T12:18:08.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 10 librados: watch_flush enter 2023-10-30T12:18:08.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 10 librados: watch_flush exit 2023-10-30T12:18:08.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 20 client.4402.objecter shutdown clearing up homeless session... 2023-10-30T12:18:08.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 10 client.4402.objecter successfully canceled tick 2023-10-30T12:18:08.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 -- 172.21.15.148:0/2535559766 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd78806c7c0 msgr2=0x7fd78806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 --2- 172.21.15.148:0/2535559766 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd78806c7c0 0x7fd78806ec80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd78c004700 tx=0x7fd78c008040 comp rx=0 tx=0).stop 2023-10-30T12:18:08.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 10 monclient: shutdown 2023-10-30T12:18:08.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:08.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 -- 172.21.15.148:0/2535559766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c12e750 msgr2=0x7fd79c13c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 --2- 172.21.15.148:0/2535559766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c12e750 0x7fd79c13c1d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd79000c700 tx=0x7fd79000cbc0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 -- 172.21.15.148:0/2535559766 shutdown_connections 2023-10-30T12:18:08.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 --2- 172.21.15.148:0/2535559766 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd78806c7c0 0x7fd78806ec80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 --2- 172.21.15.148:0/2535559766 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fd79c14ebb0 0x7fd79c14cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 --2- 172.21.15.148:0/2535559766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c12e750 0x7fd79c13c1d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 --2- 172.21.15.148:0/2535559766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd79c12e370 0x7fd79c13bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 -- 172.21.15.148:0/2535559766 >> 172.21.15.148:0/2535559766 conn(0x7fd79c0b3260 msgr2=0x7fd79c071430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:08.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 -- 172.21.15.148:0/2535559766 shutdown_connections 2023-10-30T12:18:08.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.189+0000 7fd7a14af640 1 -- 172.21.15.148:0/2535559766 wait complete. 2023-10-30T12:18:08.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.190+0000 7fd7a14af640 1 librados: shutdown 2023-10-30T12:18:08.202 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.4 2023-10-30T12:18:08.203 DEBUG:teuthology.parallel:result is None 2023-10-30T12:18:08.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.312+0000 7f3260989640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:18:08.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.312+0000 7f3260989640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:18:08.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.312+0000 7f3260989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:08.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.312+0000 7f3260989640 1 -- 172.21.15.148:0/1488302932 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f325c0ae940 con 0x7f325c0b7c10 2023-10-30T12:18:08.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.316+0000 7f3252ffd640 1 -- 172.21.15.148:0/1488302932 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f324405c2a0 con 0x7f325c0b7c10 2023-10-30T12:18:08.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.316+0000 7f3252ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:18:08.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.316+0000 7f3252ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:08.319 INFO:teuthology.orchestra.run.smithi148.stdout:64424509446 2023-10-30T12:18:08.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 10 librados: watch_flush enter 2023-10-30T12:18:08.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 10 librados: watch_flush exit 2023-10-30T12:18:08.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 20 client.4408.objecter shutdown clearing up homeless session... 2023-10-30T12:18:08.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 10 client.4408.objecter successfully canceled tick 2023-10-30T12:18:08.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 -- 172.21.15.148:0/1488302932 >> v2:172.21.15.148:6800/3205090179 conn(0x7f323c06c810 msgr2=0x7f323c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 --2- 172.21.15.148:0/1488302932 >> v2:172.21.15.148:6800/3205090179 conn(0x7f323c06c810 0x7f323c06ecd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f324c036000 tx=0x7f324c030000 comp rx=0 tx=0).stop 2023-10-30T12:18:08.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 10 monclient: shutdown 2023-10-30T12:18:08.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:08.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 -- 172.21.15.148:0/1488302932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f325c0b7c10 msgr2=0x7f325c1c83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 --2- 172.21.15.148:0/1488302932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f325c0b7c10 0x7f325c1c83e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f324400b410 tx=0x7f324400b8d0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 -- 172.21.15.148:0/1488302932 shutdown_connections 2023-10-30T12:18:08.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 --2- 172.21.15.148:0/1488302932 >> v2:172.21.15.148:6800/3205090179 conn(0x7f323c06c810 0x7f323c06ecd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 --2- 172.21.15.148:0/1488302932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f325c141940 0x7f325c1c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 --2- 172.21.15.148:0/1488302932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f325c0b7c10 0x7f325c1c83e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 --2- 172.21.15.148:0/1488302932 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f325c0b1c10 0x7f325c0ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 -- 172.21.15.148:0/1488302932 >> 172.21.15.148:0/1488302932 conn(0x7f325c06c660 msgr2=0x7f325c13dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:08.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 -- 172.21.15.148:0/1488302932 shutdown_connections 2023-10-30T12:18:08.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.318+0000 7f3260989640 1 -- 172.21.15.148:0/1488302932 wait complete. 2023-10-30T12:18:08.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.319+0000 7f3260989640 1 librados: shutdown 2023-10-30T12:18:08.331 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.2 2023-10-30T12:18:08.331 DEBUG:teuthology.parallel:result is None 2023-10-30T12:18:08.331 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T12:18:08.332 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:18:08.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.429+0000 7f65af66f640 10 monclient: get_monmap_and_config 2023-10-30T12:18:08.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.429+0000 7f65af66f640 10 monclient: build_initial_monmap 2023-10-30T12:18:08.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.429+0000 7f65af66f640 10 monclient: monmap: 2023-10-30T12:18:08.431 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:08.432 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:08.432 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:08.430758+0000 2023-10-30T12:18:08.432 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:08.430758+0000 2023-10-30T12:18:08.432 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:08.432 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:08.433 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:08.433 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:08.433 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:08.433 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:08.433 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:08.433 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:08.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.430+0000 7f65af66f640 1 Processor -- start 2023-10-30T12:18:08.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.430+0000 7f65af66f640 1 -- start start 2023-10-30T12:18:08.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.430+0000 7f65af66f640 10 monclient: init 2023-10-30T12:18:08.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.430+0000 7f65af66f640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:08.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.430+0000 7f65af66f640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T12:18:08.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.430+0000 7f65af66f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f65a80b7c10 0x7f65a80a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:08.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.430+0000 7f65af66f640 10 monclient(hunting): picked mon.noname-c con 0x7f65a80b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:08.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65af66f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65a8141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:08.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65af66f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f65a813e730 0x7f65a813eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:08.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65af66f640 10 monclient(hunting): picked mon.noname-a con 0x7f65a813e730 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:08.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65af66f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a80a9350 con 0x7f65a8141940 2023-10-30T12:18:08.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65af66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:08.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65af66f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a80670a0 con 0x7f65a813e730 2023-10-30T12:18:08.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65af66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:08.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65af66f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a80b7ff0 con 0x7f65a80b7c10 2023-10-30T12:18:08.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65af66f640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:08.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65af66f640 10 monclient(hunting): authenticate will time out at 1580.635929s 2023-10-30T12:18:08.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65adc0c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f65a813e730 0x7f65a813eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:08.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65ad40b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f65a80b7c10 0x7f65a80a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:08.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65adc0c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f65a813e730 0x7f65a813eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60198/0 (socket says 172.21.15.148:60198) 2023-10-30T12:18:08.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65adc0c640 1 -- 172.21.15.148:0/815341132 learned_addr learned my addr 172.21.15.148:0/815341132 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:08.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65adc0c640 10 monclient(hunting): get_auth_request con 0x7f65a813e730 auth_method 0 2023-10-30T12:18:08.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65adc0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:08.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65adc0c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:08.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.431+0000 7f65adc0c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:08.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.432+0000 7f65ad40b640 10 monclient(hunting): get_auth_request con 0x7f65a80b7c10 auth_method 0 2023-10-30T12:18:08.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.432+0000 7f65ad40b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:08.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.432+0000 7f65ad40b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:08.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.432+0000 7f65ad40b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:08.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.432+0000 7f65adc0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:08.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.432+0000 7f65adc0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:08.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.432+0000 7f65adc0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:08.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.432+0000 7f65ad40b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:08.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.432+0000 7f65ad40b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:08.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.432+0000 7f65ad40b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:08.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.434+0000 7f65adc0c640 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2023-10-30T12:18:08.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.434+0000 7f65adc0c640 1 -- 172.21.15.148:0/815341132 >> v2:172.21.15.150:3300/0 conn(0x7f65a80b7c10 msgr2=0x7f65a80a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:08.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.434+0000 7f65adc0c640 1 --2- 172.21.15.148:0/815341132 >> v2:172.21.15.150:3300/0 conn(0x7f65a80b7c10 0x7f65a80a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.434+0000 7f65adc0c640 1 -- 172.21.15.148:0/815341132 >> v1:172.21.15.150:6789/0 conn(0x7f65a8141940 legacy=0x7f65a8141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.434+0000 7f65adc0c640 10 monclient: _finish_hunting 0 2023-10-30T12:18:08.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.434+0000 7f65adc0c640 1 monclient: found mon.noname-a 2023-10-30T12:18:08.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.434+0000 7f65adc0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:08.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.434+0000 7f65adc0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:08.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.434+0000 7f65adc0c640 1 -- 172.21.15.148:0/815341132 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a80aa6a0 con 0x7f65a813e730 2023-10-30T12:18:08.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.436+0000 7f65adc0c640 1 --2- 172.21.15.148:0/815341132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65a813e730 0x7f65a813eb80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f65a80b4d80 tx=0x7f65a002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40122541ce7ae1c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:08.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 1 -- 172.21.15.148:0/815341132 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a002ca80 con 0x7f65a813e730 2023-10-30T12:18:08.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:08.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:08.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 10 monclient: dump: 2023-10-30T12:18:08.449 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:08.449 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:08.449 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:08.449 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:08.449 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:08.450 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:08.450 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:08.450 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:08.450 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:08.450 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:08.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 10 monclient: _finish_auth 0 2023-10-30T12:18:08.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:08.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:08.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 1 -- 172.21.15.148:0/815341132 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a002cbf0 con 0x7f65a813e730 2023-10-30T12:18:08.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:08.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 1 -- 172.21.15.148:0/815341132 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a00343f0 con 0x7f65a813e730 2023-10-30T12:18:08.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:08.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:08.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f659ffff640 10 monclient: dump: 2023-10-30T12:18:08.452 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:08.452 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:08.453 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:08.453 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:08.453 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:08.453 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:08.453 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:08.453 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:08.454 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:08.454 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:08.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f65af66f640 5 monclient: authenticate success, global_id 4374 2023-10-30T12:18:08.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f65af66f640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:08.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.439+0000 7f65af66f640 4 set_mon_vals no callback set 2023-10-30T12:18:08.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.440+0000 7f65af66f640 10 monclient: shutdown 2023-10-30T12:18:08.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.440+0000 7f65af66f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:08.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.440+0000 7f65af66f640 1 -- 172.21.15.148:0/815341132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65a813e730 msgr2=0x7f65a813eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.440+0000 7f65af66f640 1 --2- 172.21.15.148:0/815341132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65a813e730 0x7f65a813eb80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f65a80b4d80 tx=0x7f65a002bee0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.440+0000 7f65af66f640 1 -- 172.21.15.148:0/815341132 shutdown_connections 2023-10-30T12:18:08.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.440+0000 7f65af66f640 1 --2- 172.21.15.148:0/815341132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65a813e730 0x7f65a813eb80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.440+0000 7f65af66f640 1 --2- 172.21.15.148:0/815341132 >> v2:172.21.15.150:3300/0 conn(0x7f65a80b7c10 0x7f65a80a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.440+0000 7f65af66f640 1 -- 172.21.15.148:0/815341132 >> 172.21.15.148:0/815341132 conn(0x7f65a806c660 msgr2=0x7f65a80b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:08.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.440+0000 7f65af66f640 1 -- 172.21.15.148:0/815341132 shutdown_connections 2023-10-30T12:18:08.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.440+0000 7f65af66f640 1 -- 172.21.15.148:0/815341132 wait complete. 2023-10-30T12:18:08.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.441+0000 7f65af66f640 10 monclient: build_initial_monmap 2023-10-30T12:18:08.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.441+0000 7f65af66f640 10 monclient: monmap: 2023-10-30T12:18:08.457 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:08.457 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:08.457 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:08.457 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:08.457 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:08.458 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:08.458 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:08.458 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:08.458 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:08.458 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:08.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.441+0000 7f65af66f640 1 librados: starting msgr at 2023-10-30T12:18:08.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.441+0000 7f65af66f640 1 librados: starting objecter 2023-10-30T12:18:08.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.441+0000 7f65af66f640 1 Processor -- start 2023-10-30T12:18:08.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.441+0000 7f65af66f640 1 -- start start 2023-10-30T12:18:08.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.441+0000 7f65af66f640 1 librados: setting wanted keys 2023-10-30T12:18:08.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.441+0000 7f65af66f640 1 librados: calling monclient init 2023-10-30T12:18:08.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.441+0000 7f65af66f640 10 monclient: init 2023-10-30T12:18:08.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:08.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:08.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65a80b7c10 0x7f65a81da170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:08.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 10 monclient(hunting): picked mon.noname-b con 0x7f65a80b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:08.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f65a813e730 0x7f65a81da6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:08.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 10 monclient(hunting): picked mon.noname-c con 0x7f65a813e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:08.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65a8141940 0x7f65a8138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:08.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 10 monclient(hunting): picked mon.noname-a con 0x7f65a8141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:08.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:08.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a813eda0 con 0x7f65a8141940 2023-10-30T12:18:08.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:08.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a81d9d20 con 0x7f65a80b7c10 2023-10-30T12:18:08.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:08.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a80adee0 con 0x7f65a813e730 2023-10-30T12:18:08.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:08.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65af66f640 10 monclient(hunting): authenticate will time out at 1580.647117s 2023-10-30T12:18:08.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65adc0c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65a8141940 0x7f65a8138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:08.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65acc0a640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f65a813e730 0x7f65a81da6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:08.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65adc0c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65a8141940 0x7f65a8138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60210/0 (socket says 172.21.15.148:60210) 2023-10-30T12:18:08.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65adc0c640 1 -- 172.21.15.148:0/2598651042 learned_addr learned my addr 172.21.15.148:0/2598651042 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:08.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65adc0c640 10 monclient(hunting): get_auth_request con 0x7f65a8141940 auth_method 0 2023-10-30T12:18:08.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65adc0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:08.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65adc0c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:08.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.442+0000 7f65adc0c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:08.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65acc0a640 10 monclient(hunting): get_auth_request con 0x7f65a813e730 auth_method 0 2023-10-30T12:18:08.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65ad40b640 1 --2- 172.21.15.148:0/2598651042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65a80b7c10 0x7f65a81da170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:08.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65acc0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:08.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65acc0a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:08.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65acc0a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:08.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65ad40b640 10 monclient(hunting): get_auth_request con 0x7f65a80b7c10 auth_method 0 2023-10-30T12:18:08.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65ad40b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:08.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65ad40b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:08.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65ad40b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:08.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65adc0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:08.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65adc0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:08.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65adc0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:08.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65acc0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:08.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65acc0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:08.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65acc0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:08.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65ad40b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:08.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65ad40b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:08.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.443+0000 7f65ad40b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:08.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65ad40b640 10 monclient(hunting): handle_auth_done global_id 4417 payload 824 2023-10-30T12:18:08.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65ad40b640 1 -- 172.21.15.148:0/2598651042 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f65a813e730 msgr2=0x7f65a81da6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65ad40b640 1 --2- 172.21.15.148:0/2598651042 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f65a813e730 0x7f65a81da6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65ad40b640 1 -- 172.21.15.148:0/2598651042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65a8141940 msgr2=0x7f65a8138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65ad40b640 1 --2- 172.21.15.148:0/2598651042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65a8141940 0x7f65a8138ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65ad40b640 10 monclient: _finish_hunting 0 2023-10-30T12:18:08.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65ad40b640 1 monclient: found mon.noname-b 2023-10-30T12:18:08.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65ad40b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:08.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65ad40b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:08.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65ad40b640 1 -- 172.21.15.148:0/2598651042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a80aea70 con 0x7f65a80b7c10 2023-10-30T12:18:08.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.446+0000 7f65acc0a640 1 --2- 172.21.15.148:0/2598651042 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f65a813e730 0x7f65a81da6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:08.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.448+0000 7f65ad40b640 1 --2- 172.21.15.148:0/2598651042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65a80b7c10 0x7f65a81da170 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f65a400e850 tx=0x7f65a400ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:08.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.448+0000 7f659dffb640 10 client.?.objecter ms_handle_connect 0x7f65a80b7c10 2023-10-30T12:18:08.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.448+0000 7f659dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:08.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.451+0000 7f659dffb640 1 -- 172.21.15.148:0/2598651042 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a400c940 con 0x7f65a80b7c10 2023-10-30T12:18:08.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.451+0000 7f659dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:08.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.451+0000 7f659dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:08.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.451+0000 7f659dffb640 10 monclient: dump: 2023-10-30T12:18:08.472 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:08.472 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:08.473 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:08.473 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:08.473 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:08.473 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:08.473 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:08.474 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:08.474 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:08.474 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:08.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f659dffb640 10 monclient: _finish_auth 0 2023-10-30T12:18:08.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f659dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:08.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f659dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:08.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f659dffb640 1 -- 172.21.15.148:0/2598651042 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a400d040 con 0x7f65a80b7c10 2023-10-30T12:18:08.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f65af66f640 5 monclient: authenticate success, global_id 4417 2023-10-30T12:18:08.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f659dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:08.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f659dffb640 1 -- 172.21.15.148:0/2598651042 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a400ccd0 con 0x7f65a80b7c10 2023-10-30T12:18:08.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f65af66f640 10 monclient: _renew_subs 2023-10-30T12:18:08.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f65af66f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:08.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f65af66f640 1 -- 172.21.15.148:0/2598651042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a80ac0e0 con 0x7f65a80b7c10 2023-10-30T12:18:08.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f659f7fe640 4 set_mon_vals no callback set 2023-10-30T12:18:08.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f659dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:08.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f659dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:08.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f659dffb640 10 monclient: dump: 2023-10-30T12:18:08.477 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:08.477 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:08.477 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:08.477 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:08.477 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:08.478 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:08.478 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:08.478 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:08.478 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:08.478 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:08.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f65af66f640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:08.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f65af66f640 10 monclient: _renew_subs 2023-10-30T12:18:08.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f65af66f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:08.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f65af66f640 1 -- 172.21.15.148:0/2598651042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a81391e0 con 0x7f65a80b7c10 2023-10-30T12:18:08.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.452+0000 7f65af66f640 1 librados: init done 2023-10-30T12:18:08.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.453+0000 7f65af66f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:08.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.453+0000 7f65af66f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:08.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.453+0000 7f65af66f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:08.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.453+0000 7f65af66f640 1 -- 172.21.15.148:0/2598651042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a80aa350 con 0x7f65a80b7c10 2023-10-30T12:18:08.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.470+0000 7f659dffb640 1 -- 172.21.15.148:0/2598651042 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f65a4013470 con 0x7f65a80b7c10 2023-10-30T12:18:08.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.470+0000 7f659dffb640 1 --2- 172.21.15.148:0/2598651042 >> v2:172.21.15.148:6800/3205090179 conn(0x7f658c06c7f0 0x7f658c06ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:08.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.470+0000 7f65acc0a640 1 --2- 172.21.15.148:0/2598651042 >> v2:172.21.15.148:6800/3205090179 conn(0x7f658c06c7f0 0x7f658c06ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:08.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.470+0000 7f65acc0a640 10 monclient: get_auth_request con 0x7f658c06c7f0 auth_method 0 2023-10-30T12:18:08.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.470+0000 7f659dffb640 10 client.4417.objecter ms_handle_connect 0x7f658c06c7f0 2023-10-30T12:18:08.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.470+0000 7f65acc0a640 1 --2- 172.21.15.148:0/2598651042 >> v2:172.21.15.148:6800/3205090179 conn(0x7f658c06c7f0 0x7f658c06ecb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65a80ae750 tx=0x7f6598009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:08.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.472+0000 7f659dffb640 1 -- 172.21.15.148:0/2598651042 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f65a4058650 con 0x7f65a80b7c10 2023-10-30T12:18:08.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.472+0000 7f659dffb640 10 client.4417.objecter ms_dispatch 0x7f65a8000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:08.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.472+0000 7f659dffb640 3 client.4417.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:08.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.472+0000 7f659dffb640 3 client.4417.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:08.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.472+0000 7f659dffb640 20 client.4417.objecter dump_active .. 0 homeless 2023-10-30T12:18:08.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.709+0000 7f659dffb640 1 -- 172.21.15.148:0/2598651042 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65a400ce40 con 0x7f65a80b7c10 2023-10-30T12:18:08.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.709+0000 7f659dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:08.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.709+0000 7f659dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:08.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.871+0000 7f65af66f640 1 -- 172.21.15.148:0/2598651042 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65a80ac2e0 con 0x7f658c06c7f0 2023-10-30T12:18:08.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.873+0000 7f659dffb640 1 -- 172.21.15.148:0/2598651042 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47955 (secure 0 0 0) 0x7f65a80ac2e0 con 0x7f658c06c7f0 2023-10-30T12:18:08.875 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:18:08.876 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T12:18:08.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.875+0000 7f65af66f640 10 librados: watch_flush enter 2023-10-30T12:18:08.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.875+0000 7f65af66f640 10 librados: watch_flush exit 2023-10-30T12:18:08.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.875+0000 7f65af66f640 20 client.4417.objecter shutdown clearing up homeless session... 2023-10-30T12:18:08.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.875+0000 7f65af66f640 10 client.4417.objecter successfully canceled tick 2023-10-30T12:18:08.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.875+0000 7f65af66f640 1 -- 172.21.15.148:0/2598651042 >> v2:172.21.15.148:6800/3205090179 conn(0x7f658c06c7f0 msgr2=0x7f658c06ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.875+0000 7f65af66f640 1 --2- 172.21.15.148:0/2598651042 >> v2:172.21.15.148:6800/3205090179 conn(0x7f658c06c7f0 0x7f658c06ecb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65a80ae750 tx=0x7f6598009040 comp rx=0 tx=0).stop 2023-10-30T12:18:08.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.875+0000 7f65af66f640 10 monclient: shutdown 2023-10-30T12:18:08.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.875+0000 7f65af66f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:08.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.875+0000 7f65af66f640 1 -- 172.21.15.148:0/2598651042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65a80b7c10 msgr2=0x7f65a81da170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:08.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.875+0000 7f65af66f640 1 --2- 172.21.15.148:0/2598651042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65a80b7c10 0x7f65a81da170 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f65a400e850 tx=0x7f65a400ed10 comp rx=0 tx=0).stop 2023-10-30T12:18:08.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.876+0000 7f65af66f640 1 -- 172.21.15.148:0/2598651042 shutdown_connections 2023-10-30T12:18:08.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.876+0000 7f65af66f640 1 --2- 172.21.15.148:0/2598651042 >> v2:172.21.15.148:6800/3205090179 conn(0x7f658c06c7f0 0x7f658c06ecb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.876+0000 7f65af66f640 1 --2- 172.21.15.148:0/2598651042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65a8141940 0x7f65a8138ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.876+0000 7f65af66f640 1 --2- 172.21.15.148:0/2598651042 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f65a813e730 0x7f65a81da6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.876+0000 7f65af66f640 1 --2- 172.21.15.148:0/2598651042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65a80b7c10 0x7f65a81da170 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:08.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.876+0000 7f65af66f640 1 -- 172.21.15.148:0/2598651042 >> 172.21.15.148:0/2598651042 conn(0x7f65a806c660 msgr2=0x7f65a80b24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:08.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.876+0000 7f65af66f640 1 -- 172.21.15.148:0/2598651042 shutdown_connections 2023-10-30T12:18:08.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.876+0000 7f65af66f640 1 -- 172.21.15.148:0/2598651042 wait complete. 2023-10-30T12:18:08.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.876+0000 7f65af66f640 1 librados: shutdown 2023-10-30T12:18:08.888 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-30T12:18:06.965672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218900,"kb_used_data":2328,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755820,"statfs":{"total":773094113280,"available":772869959680,"internally_reserved":0,"allocated":2383872,"data_stored":1339636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":459280,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":92,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.314164"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.922859+0000","last_change":"2023-10-30T12:18:01.574170+0000","last_active":"2023-10-30T12:18:02.922859+0000","last_peered":"2023-10-30T12:18:02.922859+0000","last_clean":"2023-10-30T12:18:02.922859+0000","last_became_active":"2023-10-30T12:18:01.568787+0000","last_became_peered":"2023-10-30T12:18:01.568787+0000","last_unstale":"2023-10-30T12:18:02.922859+0000","last_undegraded":"2023-10-30T12:18:02.922859+0000","last_fullsized":"2023-10-30T12:18:02.922859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:12.243846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.134158+0000","last_change":"2023-10-30T12:18:01.632325+0000","last_active":"2023-10-30T12:18:02.134158+0000","last_peered":"2023-10-30T12:18:02.134158+0000","last_clean":"2023-10-30T12:18:02.134158+0000","last_became_active":"2023-10-30T12:18:01.602190+0000","last_became_peered":"2023-10-30T12:18:01.602190+0000","last_unstale":"2023-10-30T12:18:02.134158+0000","last_undegraded":"2023-10-30T12:18:02.134158+0000","last_fullsized":"2023-10-30T12:18:02.134158+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:26:41.530697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.919160+0000","last_change":"2023-10-30T12:18:01.060414+0000","last_active":"2023-10-30T12:18:02.919160+0000","last_peered":"2023-10-30T12:18:02.919160+0000","last_clean":"2023-10-30T12:18:02.919160+0000","last_became_active":"2023-10-30T12:18:01.051697+0000","last_became_peered":"2023-10-30T12:18:01.051697+0000","last_unstale":"2023-10-30T12:18:02.919160+0000","last_undegraded":"2023-10-30T12:18:02.919160+0000","last_fullsized":"2023-10-30T12:18:02.919160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:10.773828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.135456+0000","last_change":"2023-10-30T12:18:01.550154+0000","last_active":"2023-10-30T12:18:02.135456+0000","last_peered":"2023-10-30T12:18:02.135456+0000","last_clean":"2023-10-30T12:18:02.135456+0000","last_became_active":"2023-10-30T12:18:01.523269+0000","last_became_peered":"2023-10-30T12:18:01.523269+0000","last_unstale":"2023-10-30T12:18:02.135456+0000","last_undegraded":"2023-10-30T12:18:02.135456+0000","last_fullsized":"2023-10-30T12:18:02.135456+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:26.478209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.908014+0000","last_change":"2023-10-30T12:18:01.509923+0000","last_active":"2023-10-30T12:18:02.908014+0000","last_peered":"2023-10-30T12:18:02.908014+0000","last_clean":"2023-10-30T12:18:02.908014+0000","last_became_active":"2023-10-30T12:18:01.504543+0000","last_became_peered":"2023-10-30T12:18:01.504543+0000","last_unstale":"2023-10-30T12:18:02.908014+0000","last_undegraded":"2023-10-30T12:18:02.908014+0000","last_fullsized":"2023-10-30T12:18:02.908014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:39.886008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.134500+0000","last_change":"2023-10-30T12:18:00.932461+0000","last_active":"2023-10-30T12:18:02.134500+0000","last_peered":"2023-10-30T12:18:02.134500+0000","last_clean":"2023-10-30T12:18:02.134500+0000","last_became_active":"2023-10-30T12:18:00.904525+0000","last_became_peered":"2023-10-30T12:18:00.904525+0000","last_unstale":"2023-10-30T12:18:02.134500+0000","last_undegraded":"2023-10-30T12:18:02.134500+0000","last_fullsized":"2023-10-30T12:18:02.134500+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:04.244518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.895645+0000","last_change":"2023-10-30T12:18:01.030843+0000","last_active":"2023-10-30T12:18:02.895645+0000","last_peered":"2023-10-30T12:18:02.895645+0000","last_clean":"2023-10-30T12:18:02.895645+0000","last_became_active":"2023-10-30T12:18:01.003382+0000","last_became_peered":"2023-10-30T12:18:01.003382+0000","last_unstale":"2023-10-30T12:18:02.895645+0000","last_undegraded":"2023-10-30T12:18:02.895645+0000","last_fullsized":"2023-10-30T12:18:02.895645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:20.711334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.914105+0000","last_change":"2023-10-30T12:18:01.406267+0000","last_active":"2023-10-30T12:18:02.914105+0000","last_peered":"2023-10-30T12:18:02.914105+0000","last_clean":"2023-10-30T12:18:02.914105+0000","last_became_active":"2023-10-30T12:18:01.393697+0000","last_became_peered":"2023-10-30T12:18:01.393697+0000","last_unstale":"2023-10-30T12:18:02.914105+0000","last_undegraded":"2023-10-30T12:18:02.914105+0000","last_fullsized":"2023-10-30T12:18:02.914105+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:15:29.710122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'64","reported_seq":103,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:18:00.965938+0000","last_change":"2023-10-30T12:17:56.604947+0000","last_active":"2023-10-30T12:18:00.965938+0000","last_peered":"2023-10-30T12:18:00.965938+0000","last_clean":"2023-10-30T12:18:00.965938+0000","last_became_active":"2023-10-30T12:17:56.575838+0000","last_became_peered":"2023-10-30T12:17:56.575838+0000","last_unstale":"2023-10-30T12:18:00.965938+0000","last_undegraded":"2023-10-30T12:18:00.965938+0000","last_fullsized":"2023-10-30T12:18:00.965938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_clean_scrub_stamp":"2023-10-30T12:17:53.592563+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:47.623228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":67416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.927}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1270000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.361999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.372}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":65.984999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.370999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.904}]}]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":598016,"data_stored":465256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1989999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.323}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.678999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.866}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.868}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.082999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.895}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":67416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.510999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.041}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.158000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.134}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.841999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.492999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.728000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":72044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.883}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7759999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.394}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.725}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.692}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.728999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.216000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":67416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.362}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.526999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":528.31600000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":43.332000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":56.256}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.361000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":90.503}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":598016,"data_stored":465256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.757000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.273}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.393000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0980000000000008}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.149000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.425000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":67416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4290000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6669999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8380000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.309000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.023}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.522}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.305}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":196608,"data_stored":67416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0039999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.279}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.077999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.086}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.186}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.028}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.664000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:18:08.889 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:18:08.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.991+0000 7ff6a5644640 10 monclient: get_monmap_and_config 2023-10-30T12:18:08.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.991+0000 7ff6a5644640 10 monclient: build_initial_monmap 2023-10-30T12:18:08.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.991+0000 7ff6a5644640 10 monclient: monmap: 2023-10-30T12:18:08.993 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:08.993 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:08.993 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:08.992125+0000 2023-10-30T12:18:08.993 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:08.992125+0000 2023-10-30T12:18:08.994 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:08.994 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:08.994 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:08.994 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:08.994 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:08.995 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:08.995 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:08.995 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:08.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.991+0000 7ff6a5644640 1 Processor -- start 2023-10-30T12:18:08.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.991+0000 7ff6a5644640 1 -- start start 2023-10-30T12:18:08.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.991+0000 7ff6a5644640 10 monclient: init 2023-10-30T12:18:08.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:08.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-10-30T12:18:08.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6a00b7c10 0x7ff6a00a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:08.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a00b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:08.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff6a0141940 0x7ff6a0141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:08.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 10 monclient(hunting): picked mon.noname-b con 0x7ff6a0141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:08.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6a00b1c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:08.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a00a9350 con 0x7ff6a00b1c10 2023-10-30T12:18:08.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:08.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6a00670a0 con 0x7ff6a00b7c10 2023-10-30T12:18:08.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:08.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a00b7ff0 con 0x7ff6a0141940 2023-10-30T12:18:08.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:08.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:08.992+0000 7ff6a5644640 10 monclient(hunting): authenticate will time out at 1581.197198s 2023-10-30T12:18:09.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.004+0000 7ff69effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6a00b7c10 0x7ff6a00a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.005+0000 7ff69effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6a00b7c10 0x7ff6a00a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:49108/0 (socket says 172.21.15.148:49108) 2023-10-30T12:18:09.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.005+0000 7ff69effd640 1 -- 172.21.15.148:0/1045859936 learned_addr learned my addr 172.21.15.148:0/1045859936 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:09.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.005+0000 7ff69effd640 10 monclient(hunting): get_auth_request con 0x7ff6a00b7c10 auth_method 0 2023-10-30T12:18:09.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.005+0000 7ff69effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.005+0000 7ff69effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.005+0000 7ff69effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.006+0000 7ff69effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:09.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.006+0000 7ff69effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:09.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.006+0000 7ff69effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:09.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.008+0000 7ff69effd640 10 monclient(hunting): handle_auth_done global_id 4420 payload 274 2023-10-30T12:18:09.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.008+0000 7ff69effd640 1 -- 172.21.15.148:0/1045859936 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff6a0141940 msgr2=0x7ff6a0141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:09.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.008+0000 7ff69effd640 1 --2- 172.21.15.148:0/1045859936 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff6a0141940 0x7ff6a0141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.008+0000 7ff69effd640 1 -- 172.21.15.148:0/1045859936 >> v1:172.21.15.150:6789/0 conn(0x7ff6a00b1c10 legacy=0x7ff6a00b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.008+0000 7ff69effd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:09.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.008+0000 7ff69effd640 1 monclient: found mon.noname-c 2023-10-30T12:18:09.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.008+0000 7ff69effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:09.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.008+0000 7ff69effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:18:09.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.008+0000 7ff69effd640 1 -- 172.21.15.148:0/1045859936 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a00aa6a0 con 0x7ff6a00b7c10 2023-10-30T12:18:09.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.012+0000 7ff69effd640 1 --2- 172.21.15.148:0/1045859936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b7c10 0x7ff6a00a8aa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff690009870 tx=0x7ff69002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1558c489183b4226 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:09.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.016+0000 7ff69e7fc640 1 -- 172.21.15.148:0/1045859936 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff690031640 con 0x7ff6a00b7c10 2023-10-30T12:18:09.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:09.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 10 monclient: dump: 2023-10-30T12:18:09.019 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:09.020 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:09.020 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.020 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.020 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.021 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:09.021 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:09.021 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:09.021 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:09.021 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:09.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:09.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:09.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 1 -- 172.21.15.148:0/1045859936 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6900317b0 con 0x7ff6a00b7c10 2023-10-30T12:18:09.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 5 monclient: authenticate success, global_id 4420 2023-10-30T12:18:09.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:09.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:09.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 1 -- 172.21.15.148:0/1045859936 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff690031b50 con 0x7ff6a00b7c10 2023-10-30T12:18:09.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:09.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 4 set_mon_vals no callback set 2023-10-30T12:18:09.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:09.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff69e7fc640 10 monclient: dump: 2023-10-30T12:18:09.024 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:09.024 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:09.024 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:09.026 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:09.026 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 10 monclient: shutdown 2023-10-30T12:18:09.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:09.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 1 -- 172.21.15.148:0/1045859936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b7c10 msgr2=0x7ff6a00a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 1 --2- 172.21.15.148:0/1045859936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b7c10 0x7ff6a00a8aa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff690009870 tx=0x7ff69002b8f0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 1 -- 172.21.15.148:0/1045859936 shutdown_connections 2023-10-30T12:18:09.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 1 --2- 172.21.15.148:0/1045859936 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff6a0141940 0x7ff6a0141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 1 --2- 172.21.15.148:0/1045859936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b7c10 0x7ff6a00a8aa0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 1 -- 172.21.15.148:0/1045859936 >> 172.21.15.148:0/1045859936 conn(0x7ff6a006c660 msgr2=0x7ff6a00b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:09.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 1 -- 172.21.15.148:0/1045859936 shutdown_connections 2023-10-30T12:18:09.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.017+0000 7ff6a5644640 1 -- 172.21.15.148:0/1045859936 wait complete. 2023-10-30T12:18:09.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.018+0000 7ff6a5644640 10 monclient: build_initial_monmap 2023-10-30T12:18:09.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.018+0000 7ff6a5644640 10 monclient: monmap: 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.018+0000 7ff6a5644640 1 librados: starting msgr at 2023-10-30T12:18:09.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.018+0000 7ff6a5644640 1 librados: starting objecter 2023-10-30T12:18:09.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.018+0000 7ff6a5644640 1 Processor -- start 2023-10-30T12:18:09.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 1 -- start start 2023-10-30T12:18:09.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 1 librados: setting wanted keys 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 1 librados: calling monclient init 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient: init 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b1c10 0x7ff6a00b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient(hunting): picked mon.noname-b con 0x7ff6a00b1c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6a00b7c10 0x7ff6a00adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient(hunting): picked mon.noname-a con 0x7ff6a00b7c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff6a0141940 0x7ff6a00ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a0141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a0142130 con 0x7ff6a00b7c10 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a01c9220 con 0x7ff6a00b1c10 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a01c9500 con 0x7ff6a0141940 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:09.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.019+0000 7ff6a5644640 10 monclient(hunting): authenticate will time out at 1581.224570s 2023-10-30T12:18:09.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.020+0000 7ff69effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b1c10 0x7ff6a00b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.020+0000 7ff69effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b1c10 0x7ff6a00b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:49114/0 (socket says 172.21.15.148:49114) 2023-10-30T12:18:09.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.020+0000 7ff69effd640 1 -- 172.21.15.148:0/812876033 learned_addr learned my addr 172.21.15.148:0/812876033 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.020+0000 7ff69effd640 10 monclient(hunting): get_auth_request con 0x7ff6a00b1c10 auth_method 0 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.020+0000 7ff69effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.020+0000 7ff69effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.020+0000 7ff69effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.020+0000 7ff69effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.020+0000 7ff69effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.020+0000 7ff69effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.021+0000 7ff69f7fe640 1 --2- 172.21.15.148:0/812876033 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff6a0141940 0x7ff6a00ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.021+0000 7ff69f7fe640 10 monclient(hunting): get_auth_request con 0x7ff6a0141940 auth_method 0 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.021+0000 7ff69f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.021+0000 7ff69f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.021+0000 7ff69f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.022+0000 7ff69f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:09.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.022+0000 7ff69f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:09.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.022+0000 7ff69f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:09.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.023+0000 7ff69effd640 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2023-10-30T12:18:09.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.023+0000 7ff69effd640 1 -- 172.21.15.148:0/812876033 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff6a0141940 msgr2=0x7ff6a00ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.023+0000 7ff69effd640 1 --2- 172.21.15.148:0/812876033 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff6a0141940 0x7ff6a00ae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.023+0000 7ff69effd640 1 -- 172.21.15.148:0/812876033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6a00b7c10 msgr2=0x7ff6a00adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.023+0000 7ff69effd640 1 --2- 172.21.15.148:0/812876033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6a00b7c10 0x7ff6a00adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.023+0000 7ff69effd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:09.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.023+0000 7ff69effd640 1 monclient: found mon.noname-b 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.023+0000 7ff69effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.023+0000 7ff69effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.023+0000 7ff69effd640 1 -- 172.21.15.148:0/812876033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a01c7f90 con 0x7ff6a00b1c10 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.025+0000 7ff69effd640 1 --2- 172.21.15.148:0/812876033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b1c10 0x7ff6a00b0c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff690009990 tx=0x7ff6900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.025+0000 7ff697fff640 10 client.?.objecter ms_handle_connect 0x7ff6a00b1c10 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.025+0000 7ff697fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:09.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 1 -- 172.21.15.148:0/812876033 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69003ac60 con 0x7ff6a00b1c10 2023-10-30T12:18:09.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:09.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 10 monclient: dump: 2023-10-30T12:18:09.045 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:09.046 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:09.046 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.046 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.046 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.046 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:09.047 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:09.047 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:09.047 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:09.047 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 10 monclient: _finish_auth 0 2023-10-30T12:18:09.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:09.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:09.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff6a5644640 5 monclient: authenticate success, global_id 4423 2023-10-30T12:18:09.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 1 -- 172.21.15.148:0/812876033 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69003e030 con 0x7ff6a00b1c10 2023-10-30T12:18:09.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:09.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 1 -- 172.21.15.148:0/812876033 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6900443f0 con 0x7ff6a00b1c10 2023-10-30T12:18:09.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff6a5644640 10 monclient: _renew_subs 2023-10-30T12:18:09.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff6a5644640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:09.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff6a5644640 1 -- 172.21.15.148:0/812876033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a00afdd0 con 0x7ff6a00b1c10 2023-10-30T12:18:09.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff69dffb640 4 set_mon_vals no callback set 2023-10-30T12:18:09.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:09.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.028+0000 7ff697fff640 10 monclient: dump: 2023-10-30T12:18:09.050 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:09.050 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:09.050 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.051 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.051 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.051 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:09.051 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:09.051 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.029+0000 7ff6a5644640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.029+0000 7ff6a5644640 10 monclient: _renew_subs 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.029+0000 7ff6a5644640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.029+0000 7ff6a5644640 1 -- 172.21.15.148:0/812876033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a01c7ca0 con 0x7ff6a00b1c10 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.029+0000 7ff6a5644640 1 librados: init done 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.029+0000 7ff6a5644640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.029+0000 7ff6a5644640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.029+0000 7ff6a5644640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.029+0000 7ff6a5644640 1 -- 172.21.15.148:0/812876033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a00a8fa0 con 0x7ff6a00b1c10 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.047+0000 7ff697fff640 1 -- 172.21.15.148:0/812876033 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7ff69002f050 con 0x7ff6a00b1c10 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.047+0000 7ff697fff640 1 --2- 172.21.15.148:0/812876033 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff68006c810 0x7ff68006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.047+0000 7ff6967fc640 1 --2- 172.21.15.148:0/812876033 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff68006c810 0x7ff68006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.047+0000 7ff6967fc640 10 monclient: get_auth_request con 0x7ff68006c810 auth_method 0 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.048+0000 7ff697fff640 10 client.4423.objecter ms_handle_connect 0x7ff68006c810 2023-10-30T12:18:09.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.048+0000 7ff6967fc640 1 --2- 172.21.15.148:0/812876033 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff68006c810 0x7ff68006ecd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff6a01cdf70 tx=0x7ff688006bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:09.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.048+0000 7ff697fff640 1 -- 172.21.15.148:0/812876033 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7ff69007a1f0 con 0x7ff6a00b1c10 2023-10-30T12:18:09.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.048+0000 7ff697fff640 10 client.4423.objecter ms_dispatch 0x7ff6a0000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:09.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.048+0000 7ff697fff640 3 client.4423.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:09.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.048+0000 7ff697fff640 3 client.4423.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:09.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.048+0000 7ff697fff640 20 client.4423.objecter dump_active .. 0 homeless 2023-10-30T12:18:09.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.285+0000 7ff697fff640 1 -- 172.21.15.148:0/812876033 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff690079af0 con 0x7ff6a00b1c10 2023-10-30T12:18:09.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.285+0000 7ff697fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:09.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.285+0000 7ff697fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:09.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.452+0000 7ff6a5644640 1 -- 172.21.15.148:0/812876033 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6a01422a0 con 0x7ff68006c810 2023-10-30T12:18:09.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.454+0000 7ff697fff640 1 -- 172.21.15.148:0/812876033 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48059 (secure 0 0 0) 0x7ff688012390 con 0x7ff68006c810 2023-10-30T12:18:09.456 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:18:09.457 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T12:18:09.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 10 librados: watch_flush enter 2023-10-30T12:18:09.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 10 librados: watch_flush exit 2023-10-30T12:18:09.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 20 client.4423.objecter shutdown clearing up homeless session... 2023-10-30T12:18:09.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 10 client.4423.objecter successfully canceled tick 2023-10-30T12:18:09.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 -- 172.21.15.148:0/812876033 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff68006c810 msgr2=0x7ff68006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 --2- 172.21.15.148:0/812876033 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff68006c810 0x7ff68006ecd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff6a01cdf70 tx=0x7ff688006bf0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 10 monclient: shutdown 2023-10-30T12:18:09.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:09.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 -- 172.21.15.148:0/812876033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b1c10 msgr2=0x7ff6a00b0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 --2- 172.21.15.148:0/812876033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b1c10 0x7ff6a00b0c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff690009990 tx=0x7ff6900046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 -- 172.21.15.148:0/812876033 shutdown_connections 2023-10-30T12:18:09.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 --2- 172.21.15.148:0/812876033 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff68006c810 0x7ff68006ecd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 --2- 172.21.15.148:0/812876033 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff6a0141940 0x7ff6a00ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 --2- 172.21.15.148:0/812876033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6a00b7c10 0x7ff6a00adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 --2- 172.21.15.148:0/812876033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6a00b1c10 0x7ff6a00b0c80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 -- 172.21.15.148:0/812876033 >> 172.21.15.148:0/812876033 conn(0x7ff6a006c660 msgr2=0x7ff6a013c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:09.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 -- 172.21.15.148:0/812876033 shutdown_connections 2023-10-30T12:18:09.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 -- 172.21.15.148:0/812876033 wait complete. 2023-10-30T12:18:09.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.456+0000 7ff6a5644640 1 librados: shutdown 2023-10-30T12:18:09.469 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-10-30T12:18:08.966005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218924,"kb_used_data":2376,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755796,"statfs":{"total":773094113280,"available":772869935104,"internally_reserved":0,"allocated":2433024,"data_stored":1367404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":85,"apply_latency_ms":85,"commit_latency_ns":85000000,"apply_latency_ns":85000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":459280,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":106,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001317"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.922859+0000","last_change":"2023-10-30T12:18:01.574170+0000","last_active":"2023-10-30T12:18:02.922859+0000","last_peered":"2023-10-30T12:18:02.922859+0000","last_clean":"2023-10-30T12:18:02.922859+0000","last_became_active":"2023-10-30T12:18:01.568787+0000","last_became_peered":"2023-10-30T12:18:01.568787+0000","last_unstale":"2023-10-30T12:18:02.922859+0000","last_undegraded":"2023-10-30T12:18:02.922859+0000","last_fullsized":"2023-10-30T12:18:02.922859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:12.243846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:04.584580+0000","last_change":"2023-10-30T12:18:01.632325+0000","last_active":"2023-10-30T12:18:04.584580+0000","last_peered":"2023-10-30T12:18:04.584580+0000","last_clean":"2023-10-30T12:18:04.584580+0000","last_became_active":"2023-10-30T12:18:01.602190+0000","last_became_peered":"2023-10-30T12:18:01.602190+0000","last_unstale":"2023-10-30T12:18:04.584580+0000","last_undegraded":"2023-10-30T12:18:04.584580+0000","last_fullsized":"2023-10-30T12:18:04.584580+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:29:50.132707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.919160+0000","last_change":"2023-10-30T12:18:01.060414+0000","last_active":"2023-10-30T12:18:02.919160+0000","last_peered":"2023-10-30T12:18:02.919160+0000","last_clean":"2023-10-30T12:18:02.919160+0000","last_became_active":"2023-10-30T12:18:01.051697+0000","last_became_peered":"2023-10-30T12:18:01.051697+0000","last_unstale":"2023-10-30T12:18:02.919160+0000","last_undegraded":"2023-10-30T12:18:02.919160+0000","last_fullsized":"2023-10-30T12:18:02.919160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:10.773828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:05.140207+0000","last_change":"2023-10-30T12:18:01.550154+0000","last_active":"2023-10-30T12:18:05.140207+0000","last_peered":"2023-10-30T12:18:05.140207+0000","last_clean":"2023-10-30T12:18:05.140207+0000","last_became_active":"2023-10-30T12:18:01.523269+0000","last_became_peered":"2023-10-30T12:18:01.523269+0000","last_unstale":"2023-10-30T12:18:05.140207+0000","last_undegraded":"2023-10-30T12:18:05.140207+0000","last_fullsized":"2023-10-30T12:18:05.140207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:08.405234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.908014+0000","last_change":"2023-10-30T12:18:01.509923+0000","last_active":"2023-10-30T12:18:02.908014+0000","last_peered":"2023-10-30T12:18:02.908014+0000","last_clean":"2023-10-30T12:18:02.908014+0000","last_became_active":"2023-10-30T12:18:01.504543+0000","last_became_peered":"2023-10-30T12:18:01.504543+0000","last_unstale":"2023-10-30T12:18:02.908014+0000","last_undegraded":"2023-10-30T12:18:02.908014+0000","last_fullsized":"2023-10-30T12:18:02.908014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:39.886008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:04.515226+0000","last_change":"2023-10-30T12:18:00.932461+0000","last_active":"2023-10-30T12:18:04.515226+0000","last_peered":"2023-10-30T12:18:04.515226+0000","last_clean":"2023-10-30T12:18:04.515226+0000","last_became_active":"2023-10-30T12:18:00.904525+0000","last_became_peered":"2023-10-30T12:18:00.904525+0000","last_unstale":"2023-10-30T12:18:04.515226+0000","last_undegraded":"2023-10-30T12:18:04.515226+0000","last_fullsized":"2023-10-30T12:18:04.515226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:42.734208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.895645+0000","last_change":"2023-10-30T12:18:01.030843+0000","last_active":"2023-10-30T12:18:02.895645+0000","last_peered":"2023-10-30T12:18:02.895645+0000","last_clean":"2023-10-30T12:18:02.895645+0000","last_became_active":"2023-10-30T12:18:01.003382+0000","last_became_peered":"2023-10-30T12:18:01.003382+0000","last_unstale":"2023-10-30T12:18:02.895645+0000","last_undegraded":"2023-10-30T12:18:02.895645+0000","last_fullsized":"2023-10-30T12:18:02.895645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:20.711334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:02.914105+0000","last_change":"2023-10-30T12:18:01.406267+0000","last_active":"2023-10-30T12:18:02.914105+0000","last_peered":"2023-10-30T12:18:02.914105+0000","last_clean":"2023-10-30T12:18:02.914105+0000","last_became_active":"2023-10-30T12:18:01.393697+0000","last_became_peered":"2023-10-30T12:18:01.393697+0000","last_unstale":"2023-10-30T12:18:02.914105+0000","last_undegraded":"2023-10-30T12:18:02.914105+0000","last_fullsized":"2023-10-30T12:18:02.914105+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:15:29.710122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":119,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-30T12:18:06.032787+0000","last_change":"2023-10-30T12:17:56.604947+0000","last_active":"2023-10-30T12:18:06.032787+0000","last_peered":"2023-10-30T12:18:06.032787+0000","last_clean":"2023-10-30T12:18:06.032787+0000","last_became_active":"2023-10-30T12:17:56.575838+0000","last_became_peered":"2023-10-30T12:17:56.575838+0000","last_unstale":"2023-10-30T12:18:06.032787+0000","last_undegraded":"2023-10-30T12:18:06.032787+0000","last_fullsized":"2023-10-30T12:18:06.032787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_clean_scrub_stamp":"2023-10-30T12:17:53.592563+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:47.623228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":72044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9989999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.280999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.91}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.408999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.571}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.841999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":606208,"data_stored":469884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.032}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.754000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.31}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.34}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.059999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.835999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":72044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9710000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2149999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.272}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.379999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.491}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.148}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.872}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":72044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.883}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7759999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.394}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.725}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.692}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.728999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.216000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":72044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7960000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.524000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.523}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.053000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.731000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.734999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.792}]}]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":606208,"data_stored":469884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.757000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.273}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.393000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0980000000000008}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.149000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.425000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":72044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":38.262999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":39.850000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.131}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.297999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":42.649999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.152000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.201000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":196608,"data_stored":67416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0039999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.279}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.077999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.086}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.186}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.028}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.664000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:18:09.470 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T12:18:09.471 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T12:18:09.471 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T12:18:09.471 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T12:18:09.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.568+0000 7f4cf99fb640 10 monclient: get_monmap_and_config 2023-10-30T12:18:09.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.568+0000 7f4cf99fb640 10 monclient: build_initial_monmap 2023-10-30T12:18:09.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.568+0000 7f4cf99fb640 10 monclient: monmap: 2023-10-30T12:18:09.571 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:09.571 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:09.571 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:18:09.569696+0000 2023-10-30T12:18:09.571 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:18:09.569696+0000 2023-10-30T12:18:09.571 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:09.572 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:09.572 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:18:09.572 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:18:09.572 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:18:09.572 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:18:09.573 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:18:09.573 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 1 Processor -- start 2023-10-30T12:18:09.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 1 -- start start 2023-10-30T12:18:09.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 10 monclient: init 2023-10-30T12:18:09.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:09.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-10-30T12:18:09.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4cf40a9210 addr v1:172.21.15.150:6789/0 2023-10-30T12:18:09.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4cf4071340 0x7f4cf40aece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 10 monclient(hunting): picked mon.noname-c con 0x7f4cf4071340 addr v2:172.21.15.150:3300/0 2023-10-30T12:18:09.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4cf40af220 0x7f4cf4148690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 10 monclient(hunting): picked mon.noname-a con 0x7f4cf40af220 addr v2:172.21.15.148:3300/0 2023-10-30T12:18:09.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf40aa630 con 0x7f4cf40a9210 2023-10-30T12:18:09.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf4067340 con 0x7f4cf40af220 2023-10-30T12:18:09.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf40aa390 con 0x7f4cf4071340 2023-10-30T12:18:09.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:09.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.569+0000 7f4cf99fb640 10 monclient(hunting): authenticate will time out at 1581.774796s 2023-10-30T12:18:09.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.570+0000 7f4cf37fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4cf40af220 0x7f4cf4148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.570+0000 7f4cf27fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4cf4071340 0x7f4cf40aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.570+0000 7f4cf37fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4cf40af220 0x7f4cf4148690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60220/0 (socket says 172.21.15.148:60220) 2023-10-30T12:18:09.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.570+0000 7f4cf37fe640 1 -- 172.21.15.148:0/1859063149 learned_addr learned my addr 172.21.15.148:0/1859063149 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:09.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.570+0000 7f4cf37fe640 10 monclient(hunting): get_auth_request con 0x7f4cf40af220 auth_method 0 2023-10-30T12:18:09.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.570+0000 7f4cf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.570+0000 7f4cf37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.570+0000 7f4cf37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.571+0000 7f4cf27fc640 10 monclient(hunting): get_auth_request con 0x7f4cf4071340 auth_method 0 2023-10-30T12:18:09.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.571+0000 7f4cf27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.571+0000 7f4cf27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.571+0000 7f4cf27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.571+0000 7f4cf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.571+0000 7f4cf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.571+0000 7f4cf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:09.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.571+0000 7f4cf27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:09.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.571+0000 7f4cf27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:09.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.571+0000 7f4cf27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:09.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.572+0000 7f4cf37fe640 10 monclient(hunting): handle_auth_done global_id 4380 payload 274 2023-10-30T12:18:09.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.572+0000 7f4cf37fe640 1 -- 172.21.15.148:0/1859063149 >> v2:172.21.15.150:3300/0 conn(0x7f4cf4071340 msgr2=0x7f4cf40aece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:09.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.572+0000 7f4cf37fe640 1 --2- 172.21.15.148:0/1859063149 >> v2:172.21.15.150:3300/0 conn(0x7f4cf4071340 0x7f4cf40aece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.572+0000 7f4cf37fe640 1 -- 172.21.15.148:0/1859063149 >> v1:172.21.15.150:6789/0 conn(0x7f4cf40a9210 legacy=0x7f4cf413b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.573+0000 7f4cf37fe640 10 monclient: _finish_hunting 0 2023-10-30T12:18:09.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.573+0000 7f4cf37fe640 1 monclient: found mon.noname-a 2023-10-30T12:18:09.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.573+0000 7f4cf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:09.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.573+0000 7f4cf37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:18:09.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.573+0000 7f4cf37fe640 1 -- 172.21.15.148:0/1859063149 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf40af670 con 0x7f4cf40af220 2023-10-30T12:18:09.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.574+0000 7f4cf37fe640 1 --2- 172.21.15.148:0/1859063149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4cf40af220 0x7f4cf4148690 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4cf406bfc0 tx=0x7f4cec02d090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74986e08345fe7cd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:09.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.578+0000 7f4cf1ffb640 1 -- 172.21.15.148:0/1859063149 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec02da30 con 0x7f4cf40af220 2023-10-30T12:18:09.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:09.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 10 monclient: dump: 2023-10-30T12:18:09.589 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:09.590 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:09.590 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.590 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.590 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.590 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:09.591 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:09.591 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:09.591 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:09.591 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:18:09.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:09.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:09.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 1 -- 172.21.15.148:0/1859063149 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec02dba0 con 0x7f4cf40af220 2023-10-30T12:18:09.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:09.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 1 -- 172.21.15.148:0/1859063149 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec008500 con 0x7f4cf40af220 2023-10-30T12:18:09.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:09.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf1ffb640 10 monclient: dump: 2023-10-30T12:18:09.593 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:09.593 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:09.593 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.594 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.594 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.594 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:09.594 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:09.594 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:09.595 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:09.595 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 5 monclient: authenticate success, global_id 4380 2023-10-30T12:18:09.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:09.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 4 set_mon_vals no callback set 2023-10-30T12:18:09.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 10 monclient: shutdown 2023-10-30T12:18:09.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:09.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 1 -- 172.21.15.148:0/1859063149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4cf40af220 msgr2=0x7f4cf4148690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 1 --2- 172.21.15.148:0/1859063149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4cf40af220 0x7f4cf4148690 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4cf406bfc0 tx=0x7f4cec02d090 comp rx=0 tx=0).stop 2023-10-30T12:18:09.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 1 -- 172.21.15.148:0/1859063149 shutdown_connections 2023-10-30T12:18:09.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 1 --2- 172.21.15.148:0/1859063149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4cf40af220 0x7f4cf4148690 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 1 --2- 172.21.15.148:0/1859063149 >> v2:172.21.15.150:3300/0 conn(0x7f4cf4071340 0x7f4cf40aece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 1 -- 172.21.15.148:0/1859063149 >> 172.21.15.148:0/1859063149 conn(0x7f4cf40b3260 msgr2=0x7f4cf40b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:09.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.579+0000 7f4cf99fb640 1 -- 172.21.15.148:0/1859063149 shutdown_connections 2023-10-30T12:18:09.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.580+0000 7f4cf99fb640 1 -- 172.21.15.148:0/1859063149 wait complete. 2023-10-30T12:18:09.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.580+0000 7f4cf99fb640 10 monclient: build_initial_monmap 2023-10-30T12:18:09.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.580+0000 7f4cf99fb640 10 monclient: monmap: 2023-10-30T12:18:09.598 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:18:09.598 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:09.598 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:18:09.598 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:18:09.599 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:09.599 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:18:09.599 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:18:09.599 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:18:09.599 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:18:09.600 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.581+0000 7f4cf99fb640 1 librados: starting msgr at 2023-10-30T12:18:09.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.581+0000 7f4cf99fb640 1 librados: starting objecter 2023-10-30T12:18:09.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.581+0000 7f4cf99fb640 1 Processor -- start 2023-10-30T12:18:09.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.581+0000 7f4cf99fb640 1 -- start start 2023-10-30T12:18:09.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.581+0000 7f4cf99fb640 1 librados: setting wanted keys 2023-10-30T12:18:09.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.581+0000 7f4cf99fb640 1 librados: calling monclient init 2023-10-30T12:18:09.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.581+0000 7f4cf99fb640 10 monclient: init 2023-10-30T12:18:09.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.581+0000 7f4cf99fb640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:09.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:09.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4cf4071340 0x7f4cf41c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 10 monclient(hunting): picked mon.noname-a con 0x7f4cf4071340 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:18:09.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4cf40a9210 0x7f4cf413e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 10 monclient(hunting): picked mon.noname-c con 0x7f4cf40a9210 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:18:09.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cf40af220 0x7f4cf413bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 10 monclient(hunting): picked mon.noname-b con 0x7f4cf40af220 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:18:09.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf40ab840 con 0x7f4cf4071340 2023-10-30T12:18:09.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf413e2a0 con 0x7f4cf40af220 2023-10-30T12:18:09.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf413c1d0 con 0x7f4cf40a9210 2023-10-30T12:18:09.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:09.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf99fb640 10 monclient(hunting): authenticate will time out at 1581.786983s 2023-10-30T12:18:09.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf27fc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4cf40a9210 0x7f4cf413e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4cf4071340 0x7f4cf41c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cf40af220 0x7f4cf413bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf27fc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4cf40a9210 0x7f4cf413e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54774/0 (socket says 172.21.15.148:54774) 2023-10-30T12:18:09.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf27fc640 1 -- 172.21.15.148:0/268445862 learned_addr learned my addr 172.21.15.148:0/268445862 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:18:09.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf27fc640 10 monclient(hunting): get_auth_request con 0x7f4cf40a9210 auth_method 0 2023-10-30T12:18:09.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf2ffd640 10 monclient(hunting): get_auth_request con 0x7f4cf4071340 auth_method 0 2023-10-30T12:18:09.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.582+0000 7f4cf2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf37fe640 10 monclient(hunting): get_auth_request con 0x7f4cf40af220 auth_method 0 2023-10-30T12:18:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:09.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:09.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:09.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:09.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:09.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:09.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:09.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:09.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.583+0000 7f4cf27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:09.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.586+0000 7f4cf37fe640 10 monclient(hunting): handle_auth_done global_id 4429 payload 824 2023-10-30T12:18:09.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.586+0000 7f4cf37fe640 1 -- 172.21.15.148:0/268445862 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4cf40a9210 msgr2=0x7f4cf413e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.586+0000 7f4cf37fe640 1 --2- 172.21.15.148:0/268445862 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4cf40a9210 0x7f4cf413e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.586+0000 7f4cf37fe640 1 -- 172.21.15.148:0/268445862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4cf4071340 msgr2=0x7f4cf41c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.586+0000 7f4cf37fe640 1 --2- 172.21.15.148:0/268445862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4cf4071340 0x7f4cf41c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.586+0000 7f4cf37fe640 10 monclient: _finish_hunting 0 2023-10-30T12:18:09.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.586+0000 7f4cf37fe640 1 monclient: found mon.noname-b 2023-10-30T12:18:09.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.586+0000 7f4cf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:09.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.586+0000 7f4cf37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:18:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.586+0000 7f4cf37fe640 1 -- 172.21.15.148:0/268445862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf41339e0 con 0x7f4cf40af220 2023-10-30T12:18:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.588+0000 7f4cf37fe640 1 --2- 172.21.15.148:0/268445862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cf40af220 0x7f4cf413bc90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4cec033000 tx=0x7f4cec004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.588+0000 7f4cdbfff640 10 client.?.objecter ms_handle_connect 0x7f4cf40af220 2023-10-30T12:18:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.588+0000 7f4cdbfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 1 -- 172.21.15.148:0/268445862 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec00a620 con 0x7f4cf40af220 2023-10-30T12:18:09.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:09.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 10 monclient: dump: 2023-10-30T12:18:09.614 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:09.614 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:09.615 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.615 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.615 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.615 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:09.615 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:09.615 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:09.616 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:09.616 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 10 monclient: _finish_auth 0 2023-10-30T12:18:09.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:09.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 1 -- 172.21.15.148:0/268445862 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec040470 con 0x7f4cf40af220 2023-10-30T12:18:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 1 -- 172.21.15.148:0/268445862 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec040870 con 0x7f4cf40af220 2023-10-30T12:18:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cdbfff640 10 monclient: dump: 2023-10-30T12:18:09.618 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:18:09.618 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:18:09.618 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.618 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:18:09.618 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.619 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:18:09.619 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:18:09.619 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:18:09.619 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:18:09.619 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:18:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cf99fb640 5 monclient: authenticate success, global_id 4429 2023-10-30T12:18:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cf99fb640 10 monclient: _renew_subs 2023-10-30T12:18:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cf99fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cf17fa640 4 set_mon_vals no callback set 2023-10-30T12:18:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cf99fb640 1 -- 172.21.15.148:0/268445862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf413a670 con 0x7f4cf40af220 2023-10-30T12:18:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cf99fb640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cf99fb640 10 monclient: _renew_subs 2023-10-30T12:18:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cf99fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cf99fb640 1 -- 172.21.15.148:0/268445862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf413ab10 con 0x7f4cf40af220 2023-10-30T12:18:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.592+0000 7f4cf99fb640 1 librados: init done 2023-10-30T12:18:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.593+0000 7f4cf99fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.593+0000 7f4cf99fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.593+0000 7f4cf99fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.593+0000 7f4cf99fb640 1 -- 172.21.15.148:0/268445862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf40ab3a0 con 0x7f4cf40af220 2023-10-30T12:18:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.610+0000 7f4cdbfff640 1 -- 172.21.15.148:0/268445862 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f4cec0409e0 con 0x7f4cf40af220 2023-10-30T12:18:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.610+0000 7f4cdbfff640 1 --2- 172.21.15.148:0/268445862 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4ce006c810 0x7f4ce006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.610+0000 7f4cf2ffd640 1 --2- 172.21.15.148:0/268445862 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4ce006c810 0x7f4ce006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.610+0000 7f4cf2ffd640 10 monclient: get_auth_request con 0x7f4ce006c810 auth_method 0 2023-10-30T12:18:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.610+0000 7f4cf2ffd640 1 --2- 172.21.15.148:0/268445862 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4ce006c810 0x7f4ce006ecd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4000f60 tx=0x7f4ce4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.610+0000 7f4cdbfff640 10 client.4429.objecter ms_handle_connect 0x7f4ce006c810 2023-10-30T12:18:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.612+0000 7f4cdbfff640 1 -- 172.21.15.148:0/268445862 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f4cec04e030 con 0x7f4cf40af220 2023-10-30T12:18:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.612+0000 7f4cdbfff640 10 client.4429.objecter ms_dispatch 0x7f4cf4000b90 osd_map(24..24 src has 1..24) v4 2023-10-30T12:18:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.612+0000 7f4cdbfff640 3 client.4429.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:18:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.612+0000 7f4cdbfff640 3 client.4429.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:18:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.612+0000 7f4cdbfff640 20 client.4429.objecter dump_active .. 0 homeless 2023-10-30T12:18:09.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.850+0000 7f4cdbfff640 1 -- 172.21.15.148:0/268445862 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4cec07b6e0 con 0x7f4cf40af220 2023-10-30T12:18:09.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.850+0000 7f4cdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:09.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:09.850+0000 7f4cdbfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:10.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.063+0000 7f4cf99fb640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-10-30T12:18:10.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.063+0000 7f4cf99fb640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-10-30T12:18:10.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.063+0000 7f4cf99fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:18:10.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.063+0000 7f4cf99fb640 1 -- 172.21.15.148:0/268445862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4cf40af670 con 0x7f4cf40af220 2023-10-30T12:18:10.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.069+0000 7f4cdbfff640 1 -- 172.21.15.148:0/268445862 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4cec07f780 con 0x7f4cf40af220 2023-10-30T12:18:10.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.069+0000 7f4cdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-10-30T12:18:10.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.069+0000 7f4cdbfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:10.071 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:18:10.072 INFO:teuthology.orchestra.run.smithi148.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T12:18:10.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 10 librados: watch_flush enter 2023-10-30T12:18:10.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 10 librados: watch_flush exit 2023-10-30T12:18:10.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 20 client.4429.objecter shutdown clearing up homeless session... 2023-10-30T12:18:10.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 10 client.4429.objecter successfully canceled tick 2023-10-30T12:18:10.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 1 -- 172.21.15.148:0/268445862 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4ce006c810 msgr2=0x7f4ce006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:10.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 1 --2- 172.21.15.148:0/268445862 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4ce006c810 0x7f4ce006ecd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4000f60 tx=0x7f4ce4009040 comp rx=0 tx=0).stop 2023-10-30T12:18:10.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 10 monclient: shutdown 2023-10-30T12:18:10.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:10.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 1 -- 172.21.15.148:0/268445862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cf40af220 msgr2=0x7f4cf413bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:10.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 1 --2- 172.21.15.148:0/268445862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cf40af220 0x7f4cf413bc90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4cec033000 tx=0x7f4cec004a40 comp rx=0 tx=0).stop 2023-10-30T12:18:10.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 1 -- 172.21.15.148:0/268445862 shutdown_connections 2023-10-30T12:18:10.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 1 --2- 172.21.15.148:0/268445862 >> v2:172.21.15.148:6800/3205090179 conn(0x7f4ce006c810 0x7f4ce006ecd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:10.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 1 --2- 172.21.15.148:0/268445862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cf40af220 0x7f4cf413bc90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:10.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 1 --2- 172.21.15.148:0/268445862 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f4cf40a9210 0x7f4cf413e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:10.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.071+0000 7f4cf99fb640 1 --2- 172.21.15.148:0/268445862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4cf4071340 0x7f4cf41c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:10.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.072+0000 7f4cf99fb640 1 -- 172.21.15.148:0/268445862 >> 172.21.15.148:0/268445862 conn(0x7f4cf40b3260 msgr2=0x7f4cf40b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:10.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.072+0000 7f4cf99fb640 1 -- 172.21.15.148:0/268445862 shutdown_connections 2023-10-30T12:18:10.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.072+0000 7f4cf99fb640 1 -- 172.21.15.148:0/268445862 wait complete. 2023-10-30T12:18:10.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:18:10.072+0000 7f4cf99fb640 1 librados: shutdown 2023-10-30T12:18:10.088 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T12:18:10.089 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T12:18:10.103 INFO:tasks.workunit:Pulling workunits from ref d0db9f2f95b9d6344950185071cace2a50d296eb 2023-10-30T12:18:10.104 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T12:18:10.104 DEBUG:teuthology.orchestra.run.smithi148:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T12:18:10.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:18:10.120 INFO:teuthology.orchestra.run.smithi148.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-30T12:18:10.120 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-30T12:18:10.175 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-30T12:18:10.176 DEBUG:teuthology.orchestra.run.smithi148:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-30T12:18:10.230 INFO:tasks.workunit:timeout=6h 2023-10-30T12:18:10.230 INFO:tasks.workunit:cleanup=True 2023-10-30T12:18:10.231 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d0db9f2f95b9d6344950185071cace2a50d296eb 2023-10-30T12:18:10.285 INFO:tasks.workunit.client.0.smithi148.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T12:18:20.260 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:01:35.334 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:20.264 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:01:35.337 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:20.311 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:01:35.358 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:20.312 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:01:35.386 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:20.313 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:01:35.387 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:20.316 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:01:35.364 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:20.320 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:01:35.394 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:20.360 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:01:35.406 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:20.366 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:01:35.440 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:40.305 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:01:55.378 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:40.306 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:01:55.380 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:40.325 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:01:55.399 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:40.356 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:01:55.429 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:40.360 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:01:55.434 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:40.361 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:01:55.408 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:40.363 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:01:55.410 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:40.378 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:01:55.451 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:40.378 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:01:55.425 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:00.321 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:02:15.394 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:00.400 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:02:15.474 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:00.426 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:02:15.500 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:00.449 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:02:15.497 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:00.451 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:02:15.498 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:00.465 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:02:15.512 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:00.477 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:15.550 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:00.482 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:15.556 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:00.507 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:15.581 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:13.858 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 99% (11984/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T12:19:14.429 INFO:tasks.workunit.client.0.smithi148.stderr:Note: switching to 'd0db9f2f95b9d6344950185071cace2a50d296eb'. 2023-10-30T12:19:14.429 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T12:19:14.429 INFO:tasks.workunit.client.0.smithi148.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T12:19:14.430 INFO:tasks.workunit.client.0.smithi148.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T12:19:14.430 INFO:tasks.workunit.client.0.smithi148.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T12:19:14.430 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T12:19:14.430 INFO:tasks.workunit.client.0.smithi148.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T12:19:14.430 INFO:tasks.workunit.client.0.smithi148.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T12:19:14.431 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T12:19:14.431 INFO:tasks.workunit.client.0.smithi148.stderr: git switch -c 2023-10-30T12:19:14.431 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T12:19:14.431 INFO:tasks.workunit.client.0.smithi148.stderr:Or undo this operation with: 2023-10-30T12:19:14.431 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T12:19:14.432 INFO:tasks.workunit.client.0.smithi148.stderr: git switch - 2023-10-30T12:19:14.432 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T12:19:14.432 INFO:tasks.workunit.client.0.smithi148.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T12:19:14.432 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T12:19:14.433 INFO:tasks.workunit.client.0.smithi148.stderr:HEAD is now at d0db9f2f95b Merge pull request #54082 from pdvian/wip-qa-suites-fix 2023-10-30T12:19:14.434 DEBUG:teuthology.orchestra.run.smithi148:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T12:19:14.499 INFO:tasks.workunit.client.0.smithi148.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T12:19:14.501 INFO:tasks.workunit.client.0.smithi148.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T12:19:14.502 INFO:tasks.workunit.client.0.smithi148.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T12:19:15.198 INFO:tasks.workunit.client.0.smithi148.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T12:19:15.269 INFO:tasks.workunit.client.0.smithi148.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T12:19:15.310 INFO:tasks.workunit.client.0.smithi148.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T12:19:15.311 INFO:tasks.workunit.client.0.smithi148.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T12:19:15.312 INFO:tasks.workunit.client.0.smithi148.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T12:19:15.354 INFO:tasks.workunit.client.0.smithi148.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T12:19:15.357 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:19:15.357 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T12:19:15.412 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-10-30T12:19:15.413 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-10-30T12:19:15.414 DEBUG:teuthology.orchestra.run.smithi148:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0db9f2f95b9d6344950185071cace2a50d296eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-10-30T12:19:15.472 INFO:tasks.workunit.client.0.smithi148.stderr:+ parallel=1 2023-10-30T12:19:15.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' '' = --serial ']' 2023-10-30T12:19:15.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ color= 2023-10-30T12:19:15.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -t 1 ']' 2023-10-30T12:19:15.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-10-30T12:19:15.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ declare -A pids 2023-10-30T12:19:15.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.474 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_aio 2023-10-30T12:19:15.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_aio' 2023-10-30T12:19:15.474 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_aio 2023-10-30T12:19:15.475 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.475 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_aio 2023-10-30T12:19:15.475 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91850 2023-10-30T12:19:15.476 INFO:tasks.workunit.client.0.smithi148.stdout:test api_aio on pid 91850 2023-10-30T12:19:15.478 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_aio on pid 91850' 2023-10-30T12:19:15.478 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=91850 2023-10-30T12:19:15.479 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.479 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.479 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-10-30T12:19:15.479 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_aio_pp 2023-10-30T12:19:15.480 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_aio_pp' 2023-10-30T12:19:15.480 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_aio_pp 2023-10-30T12:19:15.480 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.480 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.480 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.481 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.481 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.481 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.481 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.481 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.481 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.482 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.482 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.482 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.482 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.482 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.483 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.483 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.483 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.483 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.483 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.484 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.484 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.484 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.484 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.484 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.484 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.485 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.485 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_aio_pp 2023-10-30T12:19:15.485 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91857 2023-10-30T12:19:15.485 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_aio_pp on pid 91857' 2023-10-30T12:19:15.486 INFO:tasks.workunit.client.0.smithi148.stdout:test api_aio_pp on pid 91857 2023-10-30T12:19:15.489 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=91857 2023-10-30T12:19:15.489 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.489 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.490 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-10-30T12:19:15.490 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_io 2023-10-30T12:19:15.490 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_io' 2023-10-30T12:19:15.490 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_io 2023-10-30T12:19:15.490 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.490 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.491 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.491 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.491 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.491 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.491 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.492 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.492 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.492 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.492 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.492 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.493 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.493 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.493 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.493 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.493 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.493 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.494 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.494 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.494 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.494 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.494 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.495 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.495 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.495 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.495 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.495 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.496 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.496 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.496 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.496 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.496 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.497 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.497 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.497 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.497 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.497 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.498 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.498 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.498 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.498 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.498 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.499 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.499 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.499 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.499 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_io 2023-10-30T12:19:15.499 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91866 2023-10-30T12:19:15.500 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_io on pid 91866' 2023-10-30T12:19:15.500 INFO:tasks.workunit.client.0.smithi148.stdout:test api_io on pid 91866 2023-10-30T12:19:15.504 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=91866 2023-10-30T12:19:15.504 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.504 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-10-30T12:19:15.504 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.505 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_io_pp 2023-10-30T12:19:15.505 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_io_pp' 2023-10-30T12:19:15.505 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_io_pp 2023-10-30T12:19:15.505 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.505 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.505 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.506 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.506 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.506 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.506 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.506 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.507 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.507 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.507 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.507 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.508 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.508 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.508 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.508 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.508 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.508 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.509 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.509 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.509 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.509 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.509 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.510 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.510 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.510 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.510 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.510 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.510 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.511 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.511 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.511 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.511 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.511 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.512 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.512 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.512 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.512 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.512 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.513 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.513 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.513 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.513 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.513 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.513 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.514 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.514 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.514 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.514 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.514 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.515 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.515 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.515 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.515 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.515 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.516 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_io_pp 2023-10-30T12:19:15.516 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91878 2023-10-30T12:19:15.516 INFO:tasks.workunit.client.0.smithi148.stdout:test api_io_pp on pid 91878 2023-10-30T12:19:15.521 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_io_pp on pid 91878' 2023-10-30T12:19:15.521 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=91878 2023-10-30T12:19:15.521 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.522 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.522 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-10-30T12:19:15.522 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_asio 2023-10-30T12:19:15.522 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_asio' 2023-10-30T12:19:15.522 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_asio 2023-10-30T12:19:15.523 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.523 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.523 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.523 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.523 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.524 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.524 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.524 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.524 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.524 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.525 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.525 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.525 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.525 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.525 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.525 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.526 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.526 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.526 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.526 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.526 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.527 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.527 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.527 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.527 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.527 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.528 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.528 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.528 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.528 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.528 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.528 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.529 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.529 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.529 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.529 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.529 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.530 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.530 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.530 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.530 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.530 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.530 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.531 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.531 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.531 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.531 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.531 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.532 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.532 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.532 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.532 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.532 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.532 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.533 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.533 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.533 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.533 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.533 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.534 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.534 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.534 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.534 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.534 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.535 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.535 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.535 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.535 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.535 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.536 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.536 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.536 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.536 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.536 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_asio 2023-10-30T12:19:15.536 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91893 2023-10-30T12:19:15.537 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.537 INFO:tasks.workunit.client.0.smithi148.stdout:test api_asio on pid 91893 2023-10-30T12:19:15.546 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_asio on pid 91893' 2023-10-30T12:19:15.546 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.547 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=91893 2023-10-30T12:19:15.547 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.547 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.547 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-10-30T12:19:15.547 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.548 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.548 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.548 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_list 2023-10-30T12:19:15.548 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.548 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.549 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_list' 2023-10-30T12:19:15.549 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.549 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.549 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.549 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.550 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.550 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.550 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.550 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.550 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.551 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.551 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.551 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.551 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.551 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.551 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.552 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.552 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_list 2023-10-30T12:19:15.552 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.552 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.552 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.553 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.553 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.553 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.553 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.553 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.554 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.554 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.554 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.554 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.554 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.555 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.555 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.555 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.555 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.555 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.555 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.557 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.557 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.557 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.557 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.557 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.559 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.559 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.559 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.560 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.560 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.560 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.560 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.560 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.561 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.561 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.561 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.561 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.561 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.561 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.562 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.562 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.562 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.562 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.562 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.563 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.563 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.563 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.563 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.563 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.564 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.564 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.564 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.564 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.564 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.564 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.565 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.565 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.565 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.565 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.565 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.566 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.566 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.566 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.566 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.566 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.567 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.567 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.567 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/91850/exe 2023-10-30T12:19:15.567 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.567 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.567 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.568 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.568 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.568 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_list 2023-10-30T12:19:15.568 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.568 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91910 2023-10-30T12:19:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_list on pid 91910' 2023-10-30T12:19:15.569 INFO:tasks.workunit.client.0.smithi148.stdout:test api_list on pid 91910 2023-10-30T12:19:15.576 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=91910 2023-10-30T12:19:15.576 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.576 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.577 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-10-30T12:19:15.577 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.577 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_lock 2023-10-30T12:19:15.577 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.577 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.578 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.578 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_lock' 2023-10-30T12:19:15.578 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.578 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.578 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.579 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.579 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.579 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.579 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_lock 2023-10-30T12:19:15.579 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.580 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.580 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.580 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.580 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.580 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.581 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.581 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.581 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.581 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.581 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.582 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.582 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.582 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.582 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.582 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.583 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.583 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.583 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.583 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.583 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.584 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.584 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.584 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.584 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.584 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.585 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.585 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.585 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.585 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.585 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.585 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.586 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.586 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.586 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.586 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.586 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.587 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.587 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.587 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.587 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.587 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.588 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.588 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.588 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.588 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.588 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.588 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.589 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.589 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.589 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.589 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.589 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.590 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.590 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.590 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.590 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.590 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.591 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.591 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.591 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.591 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.591 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.592 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.592 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.592 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.592 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.592 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.592 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.593 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.593 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.593 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.593 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.593 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.594 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.594 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.594 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.594 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_aio 2023-10-30T12:19:15.594 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.595 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.595 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_aio: /' 2023-10-30T12:19:15.595 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.595 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.595 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.595 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.596 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/91857/exe 2023-10-30T12:19:15.596 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.596 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_aio.log 2023-10-30T12:19:15.596 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.596 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.597 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.597 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.597 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.597 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.597 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_lock 2023-10-30T12:19:15.599 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91930 2023-10-30T12:19:15.599 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_lock on pid 91930' 2023-10-30T12:19:15.599 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-10-30T12:19:15.600 INFO:tasks.workunit.client.0.smithi148.stdout:test api_lock on pid 91930 2023-10-30T12:19:15.613 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=91930 2023-10-30T12:19:15.613 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.613 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.614 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.614 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_lock_pp 2023-10-30T12:19:15.614 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.614 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.614 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.615 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.615 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.615 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.615 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_lock_pp' 2023-10-30T12:19:15.615 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.616 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.616 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.616 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.616 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.616 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.617 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.617 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.617 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.617 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.617 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.617 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.618 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.618 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.618 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.618 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.618 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.619 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.619 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.619 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_lock_pp 2023-10-30T12:19:15.619 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.619 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.620 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.620 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.620 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.620 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.620 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.620 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.621 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.621 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.621 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-10-30T12:19:15.621 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.621 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.622 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.622 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.622 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.622 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.622 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.623 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.623 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_aio_pp 2023-10-30T12:19:15.623 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.623 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.623 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.623 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.624 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.624 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.624 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.624 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.624 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.625 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_aio_pp: /' 2023-10-30T12:19:15.625 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.625 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.625 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.625 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.626 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.626 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.626 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.626 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.626 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.627 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.627 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.627 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.627 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.627 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.627 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.628 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.628 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.628 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.628 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.628 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.629 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.629 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.629 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.629 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.629 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.630 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.630 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.630 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.630 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.630 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.631 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.631 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.631 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.631 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.631 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.631 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.632 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.632 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.632 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.632 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.632 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.633 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.633 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.633 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.633 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.633 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.634 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.634 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.634 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.634 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.634 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.635 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.635 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.635 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.635 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.635 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.635 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.636 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.636 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.636 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.636 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.636 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.637 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.637 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/91866/exe 2023-10-30T12:19:15.637 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.637 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.637 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.637 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.638 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.638 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.638 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.638 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.638 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.639 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.639 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.639 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.639 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.639 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.640 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.640 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.640 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.640 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.640 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.641 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.641 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.641 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.641 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.641 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.641 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.643 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.643 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.643 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.643 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.643 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.643 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.644 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.644 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.644 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.644 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.644 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.645 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.645 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.645 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.645 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.645 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.645 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.646 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.646 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.646 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.646 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.646 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.647 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.647 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.647 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.647 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.647 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.648 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.648 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.648 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.648 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.648 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.648 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.649 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.649 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.649 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.649 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.649 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.650 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.650 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.650 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.650 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.650 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.651 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.651 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.651 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.651 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_lock_pp 2023-10-30T12:19:15.651 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.652 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.652 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.652 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91957 2023-10-30T12:19:15.652 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_lock_pp on pid 91957' 2023-10-30T12:19:15.652 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_io 2023-10-30T12:19:15.653 INFO:tasks.workunit.client.0.smithi148.stdout:test api_lock_pp on pid 91957 2023-10-30T12:19:15.659 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=91957 2023-10-30T12:19:15.659 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.659 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.660 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_io.log 2023-10-30T12:19:15.660 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/91878/exe 2023-10-30T12:19:15.660 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.660 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.660 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_misc 2023-10-30T12:19:15.661 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.661 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.661 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.661 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_misc' 2023-10-30T12:19:15.661 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-10-30T12:19:15.661 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.662 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.662 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_io: /' 2023-10-30T12:19:15.662 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.662 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.662 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.663 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.663 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.663 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.663 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.663 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.664 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.664 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.664 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.664 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.664 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.665 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.665 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.665 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.665 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.665 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.666 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.666 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.666 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.666 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.666 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.667 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.667 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.667 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.667 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.667 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.667 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.668 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.668 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.668 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.668 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.668 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.669 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.669 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.669 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.669 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.669 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.670 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.670 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.670 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.670 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.670 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.671 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.671 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.671 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.671 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.671 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.671 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.672 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.672 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-10-30T12:19:15.672 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.672 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.672 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.673 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.673 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_io_pp 2023-10-30T12:19:15.673 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_misc 2023-10-30T12:19:15.673 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.673 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.674 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.674 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.674 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.674 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.674 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.675 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.675 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.675 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.675 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_io_pp: /' 2023-10-30T12:19:15.675 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.675 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_misc 2023-10-30T12:19:15.676 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91977 2023-10-30T12:19:15.676 INFO:tasks.workunit.client.0.smithi148.stdout:test api_misc on pid 91977 2023-10-30T12:19:15.698 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_misc on pid 91977' 2023-10-30T12:19:15.698 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=91977 2023-10-30T12:19:15.698 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.698 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.698 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.699 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.699 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.699 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.699 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.699 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.700 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.700 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.700 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.700 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.700 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.701 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.701 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.701 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.701 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.701 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.701 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.702 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.702 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.702 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.702 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.702 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.703 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.703 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.703 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-10-30T12:19:15.703 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.703 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.704 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.704 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.704 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.704 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.704 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.704 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.705 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.705 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.705 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.705 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.705 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.706 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.706 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/91893/exe 2023-10-30T12:19:15.706 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.706 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.706 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.707 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.707 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.707 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.707 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.707 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.707 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.708 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.708 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.708 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.708 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.708 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.709 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/91910/exe 2023-10-30T12:19:15.709 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.709 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.709 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.709 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.709 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.710 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.710 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.710 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.710 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.710 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.711 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.711 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.711 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.711 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.711 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.711 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.712 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.712 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.712 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.712 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.712 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.713 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.713 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.713 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.713 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.713 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.713 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.714 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.714 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.714 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.714 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.714 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_asio.log 2023-10-30T12:19:15.715 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.715 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.715 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.715 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.715 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.716 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.716 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.716 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.716 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.716 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.717 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_list 2023-10-30T12:19:15.717 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.717 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_misc_pp 2023-10-30T12:19:15.717 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.717 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.717 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.718 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.718 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.718 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.718 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.718 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.719 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.719 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.719 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.719 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_misc_pp' 2023-10-30T12:19:15.719 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.719 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.720 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.720 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.720 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.720 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_asio: /' 2023-10-30T12:19:15.720 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.721 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.721 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.721 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.721 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.721 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.721 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.722 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_misc_pp 2023-10-30T12:19:15.722 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.722 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.722 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.722 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.723 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.723 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.723 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.723 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.723 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.724 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.724 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.724 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.724 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.724 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.724 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.725 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.725 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.725 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.725 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.725 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.726 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.726 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.726 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.726 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.726 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.727 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.727 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.727 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.727 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.727 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.728 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.728 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.728 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.728 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.728 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.728 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.729 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.729 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.729 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.729 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.729 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.730 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.730 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.730 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.730 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.730 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.731 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.731 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.731 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.731 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.731 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.731 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.732 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.732 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.732 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.732 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.732 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.733 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.733 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.733 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.733 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.733 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.734 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.734 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.734 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.734 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.734 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.735 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/91930/exe 2023-10-30T12:19:15.735 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.735 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.735 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.735 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.735 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_list: /' 2023-10-30T12:19:15.736 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.736 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.736 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.736 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.736 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.736 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.737 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.737 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.737 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.737 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_asio 2023-10-30T12:19:15.737 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.738 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.738 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.738 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.738 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.738 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.739 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.739 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.739 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.739 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.739 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.739 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.740 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.740 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.740 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.740 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.740 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.741 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.741 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.741 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.741 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.741 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.741 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.742 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.742 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_lock 2023-10-30T12:19:15.742 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.742 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.742 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.743 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.743 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.743 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.743 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.743 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.744 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.744 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.744 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.744 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.744 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.744 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.745 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.745 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.745 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.745 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.745 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.746 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.746 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.746 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_lock: /' 2023-10-30T12:19:15.746 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/91957/exe 2023-10-30T12:19:15.746 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.747 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.747 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.747 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.747 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.749 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.749 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.749 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.750 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.750 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.750 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.750 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_list.log 2023-10-30T12:19:15.750 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-10-30T12:19:15.751 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.751 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_misc_pp 2023-10-30T12:19:15.751 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_lock_pp 2023-10-30T12:19:15.751 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92030 2023-10-30T12:19:15.751 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-10-30T12:19:15.752 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_lock_pp: /' 2023-10-30T12:19:15.752 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_misc_pp on pid 92030' 2023-10-30T12:19:15.752 INFO:tasks.workunit.client.0.smithi148.stdout:test api_misc_pp on pid 92030 2023-10-30T12:19:15.755 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92030 2023-10-30T12:19:15.755 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.755 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.755 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_tier_pp 2023-10-30T12:19:15.755 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_tier_pp' 2023-10-30T12:19:15.756 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_lock.log 2023-10-30T12:19:15.756 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.756 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.756 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.756 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.756 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.757 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.757 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.757 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.757 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.757 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_tier_pp 2023-10-30T12:19:15.758 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.758 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.758 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.758 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.758 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.759 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.759 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.759 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_tier_pp 2023-10-30T12:19:15.759 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92048 2023-10-30T12:19:15.759 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_tier_pp on pid 92048' 2023-10-30T12:19:15.760 INFO:tasks.workunit.client.0.smithi148.stdout:test api_tier_pp on pid 92048 2023-10-30T12:19:15.766 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92048 2023-10-30T12:19:15.766 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.766 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.766 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_pool 2023-10-30T12:19:15.767 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_pool' 2023-10-30T12:19:15.767 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.767 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.767 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.767 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-10-30T12:19:15.768 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.768 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.768 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.768 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.768 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_pool 2023-10-30T12:19:15.769 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.769 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.769 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.769 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.769 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.769 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.770 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.770 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.770 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.770 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.770 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.771 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.771 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.771 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.771 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.771 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.771 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.772 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.772 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.772 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.772 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.772 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.773 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.773 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.773 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.773 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.773 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.773 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.774 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.774 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.774 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.774 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.774 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.775 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.775 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.775 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.775 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.775 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.776 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.776 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.776 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.776 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.776 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.776 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.777 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.777 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.777 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.777 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.777 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.778 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.778 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.778 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.778 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.778 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.779 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.779 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_pool 2023-10-30T12:19:15.779 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92071 2023-10-30T12:19:15.779 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.779 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_pool on pid 92071' 2023-10-30T12:19:15.780 INFO:tasks.workunit.client.0.smithi148.stdout:test api_pool on pid 92071 2023-10-30T12:19:15.785 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92071 2023-10-30T12:19:15.785 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.785 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.785 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.786 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.786 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.786 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.786 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.786 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.787 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.787 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.787 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.787 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.787 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.789 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.789 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.789 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.789 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-10-30T12:19:15.789 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.790 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.790 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.790 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.790 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.790 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.791 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.791 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.791 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.791 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.791 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.791 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.792 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_snapshots 2023-10-30T12:19:15.792 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_snapshots' 2023-10-30T12:19:15.792 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.792 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.792 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.793 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.793 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.793 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.793 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.793 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.794 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.794 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.794 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.794 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.794 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.795 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.795 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.795 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.795 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.795 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.796 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.796 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.796 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.796 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.796 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_snapshots 2023-10-30T12:19:15.796 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/91977/exe 2023-10-30T12:19:15.797 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.797 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_snapshots 2023-10-30T12:19:15.797 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.797 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.797 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.798 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92094 2023-10-30T12:19:15.798 INFO:tasks.workunit.client.0.smithi148.stdout:test api_snapshots on pid 92094 2023-10-30T12:19:15.810 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_snapshots on pid 92094' 2023-10-30T12:19:15.811 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92094 2023-10-30T12:19:15.811 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.811 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.811 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.811 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.812 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.812 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-10-30T12:19:15.812 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.812 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.812 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.812 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_snapshots_pp 2023-10-30T12:19:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_snapshots_pp' 2023-10-30T12:19:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.815 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.815 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.815 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.815 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.815 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.817 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.817 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_misc 2023-10-30T12:19:15.817 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_misc.log 2023-10-30T12:19:15.817 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.817 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.817 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.818 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.818 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.818 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.818 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.818 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.819 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.819 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.819 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.819 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.819 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.820 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.820 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.820 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.820 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.820 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.820 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.821 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.821 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.821 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.821 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.821 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.822 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.822 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.822 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_misc: /' 2023-10-30T12:19:15.822 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.822 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.823 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.823 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.823 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.823 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.823 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.823 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.824 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.824 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.824 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.824 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.824 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.825 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.825 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.825 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.825 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.825 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.825 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.826 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.826 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.826 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.826 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.827 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.827 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.827 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.827 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_snapshots_pp 2023-10-30T12:19:15.827 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.827 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.828 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.828 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.828 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.828 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.828 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.829 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.829 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.829 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.829 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.829 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.829 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.830 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.830 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.830 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.830 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.830 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.831 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.831 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.831 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.831 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.831 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.831 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.832 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.832 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.832 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.832 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.832 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.833 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.833 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.833 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.833 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.833 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.833 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.834 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.834 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.834 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.834 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.834 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.835 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.835 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.835 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.835 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.835 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.836 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.836 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.836 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.836 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.836 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.836 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.837 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.837 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.837 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.837 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.837 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.838 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.838 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.838 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.838 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.838 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.839 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.839 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.839 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.839 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.839 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.839 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.840 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.840 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.840 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.840 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_snapshots_pp 2023-10-30T12:19:15.840 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.841 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.841 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.841 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.841 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.841 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92136 2023-10-30T12:19:15.841 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-10-30T12:19:15.842 INFO:tasks.workunit.client.0.smithi148.stdout:test api_snapshots_pp on pid 92136 2023-10-30T12:19:15.854 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_snapshots_pp on pid 92136' 2023-10-30T12:19:15.854 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92136 2023-10-30T12:19:15.854 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.854 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.855 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_stat 2023-10-30T12:19:15.855 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_stat' 2023-10-30T12:19:15.855 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.855 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.855 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.855 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.856 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.856 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.856 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.856 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.856 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.857 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.857 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.857 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.857 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.857 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.858 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.858 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.858 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.858 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.858 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.858 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.859 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.859 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.859 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.859 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.859 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.860 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.860 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.860 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.860 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.860 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.861 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.861 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.861 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.861 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.861 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.861 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.862 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.862 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.862 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.862 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.862 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.863 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.863 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.863 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.863 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.863 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.863 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.864 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.864 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.864 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.864 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.864 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.865 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.865 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.865 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.865 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.865 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.865 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.866 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.866 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.866 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.866 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_stat 2023-10-30T12:19:15.866 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.867 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.867 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.867 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.867 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.867 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.868 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.868 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.868 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.868 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.868 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.868 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.869 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.869 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.869 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.869 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.869 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.870 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.870 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.870 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.870 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.870 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.871 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.871 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.871 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.871 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.871 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.871 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.872 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.872 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.872 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.872 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.872 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92030/exe 2023-10-30T12:19:15.873 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.873 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.873 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.873 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.873 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.873 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.874 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.874 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.874 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.874 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.874 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.875 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.875 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.875 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.875 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.875 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.876 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.876 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.876 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.876 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.876 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.876 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.877 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.877 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.877 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.877 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.877 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.878 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.878 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.878 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.878 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.878 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.879 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_misc_pp 2023-10-30T12:19:15.879 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.879 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.879 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.879 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.879 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.880 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.880 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.880 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.880 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.880 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.880 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.881 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.881 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.881 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.881 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.881 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.882 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.882 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.882 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.882 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.882 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.883 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.883 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.883 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.883 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.883 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.883 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.884 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.884 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.884 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.884 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.884 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.885 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.885 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.885 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.885 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-10-30T12:19:15.885 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.886 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_stat 2023-10-30T12:19:15.886 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.886 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.886 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.886 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.887 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.887 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.887 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.887 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.887 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92177 2023-10-30T12:19:15.887 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.888 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.888 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.888 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_stat on pid 92177' 2023-10-30T12:19:15.888 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.889 INFO:tasks.workunit.client.0.smithi148.stdout:test api_stat on pid 92177 2023-10-30T12:19:15.895 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.895 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92177 2023-10-30T12:19:15.896 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.896 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.896 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.896 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.896 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.897 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.897 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.897 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.897 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.897 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.897 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.898 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.898 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.898 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.898 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_misc_pp: /' 2023-10-30T12:19:15.898 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.899 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_stat_pp 2023-10-30T12:19:15.899 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-10-30T12:19:15.899 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_stat_pp' 2023-10-30T12:19:15.899 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.899 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.900 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.900 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.900 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92094/exe 2023-10-30T12:19:15.900 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.900 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.900 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.901 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.901 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.901 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.901 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.901 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_stat_pp 2023-10-30T12:19:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92048/exe 2023-10-30T12:19:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.906 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.906 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.906 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.906 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.906 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.910 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.910 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.910 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.910 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.910 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.911 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.911 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.911 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.911 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.911 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.911 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.912 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.912 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.912 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.912 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.912 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.913 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.913 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.913 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_stat_pp 2023-10-30T12:19:15.913 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.913 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.913 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92192 2023-10-30T12:19:15.914 INFO:tasks.workunit.client.0.smithi148.stdout:test api_stat_pp on pid 92192 2023-10-30T12:19:15.926 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_stat_pp on pid 92192' 2023-10-30T12:19:15.926 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92192 2023-10-30T12:19:15.926 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.926 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.927 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.927 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.927 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.927 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-10-30T12:19:15.927 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-10-30T12:19:15.927 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_tier_pp: /' 2023-10-30T12:19:15.928 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_tier_pp 2023-10-30T12:19:15.928 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.928 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.928 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.928 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.929 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.929 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_watch_notify 2023-10-30T12:19:15.929 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.929 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.929 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.930 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.930 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.930 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.930 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.930 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.931 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.931 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.931 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_watch_notify' 2023-10-30T12:19:15.931 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.931 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.932 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.932 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.932 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.932 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.932 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.933 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.933 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.933 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.933 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.933 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.934 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.934 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.934 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.934 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.934 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.934 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.935 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.935 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.935 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.935 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.935 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.936 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.936 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.936 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.936 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.936 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.937 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.937 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.937 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_watch_notify 2023-10-30T12:19:15.937 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.937 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92071/exe 2023-10-30T12:19:15.937 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.938 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.938 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.938 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.938 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.938 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.939 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.939 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.939 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.939 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.939 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.940 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.940 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.940 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.940 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.940 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.940 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.941 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.941 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.941 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.941 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.941 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.942 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.942 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.942 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.942 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.942 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.943 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.943 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.943 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.943 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.943 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.943 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.944 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.944 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.944 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.944 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.944 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.945 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.945 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.945 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.945 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.945 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.946 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.946 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.946 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.946 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.946 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.947 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.947 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.947 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.947 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-10-30T12:19:15.947 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.947 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.948 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.948 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.948 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.948 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.948 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.949 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.949 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_snapshots: /' 2023-10-30T12:19:15.949 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.949 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.949 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_pool 2023-10-30T12:19:15.949 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_pool: /' 2023-10-30T12:19:15.950 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:15.950 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_pool.log 2023-10-30T12:19:15.950 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:15.950 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:15.950 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.951 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:15.951 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.951 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:15.951 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:15.951 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:15.952 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.952 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:15.952 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.952 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:15.952 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:15.953 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:15.953 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.953 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:15.953 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.953 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:15.954 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.954 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_snapshots 2023-10-30T12:19:15.954 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.954 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.954 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.958 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.958 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.958 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.959 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.959 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.959 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.959 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.959 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_watch_notify 2023-10-30T12:19:15.960 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92225 2023-10-30T12:19:15.960 INFO:tasks.workunit.client.0.smithi148.stdout:test api_watch_notify on pid 92225 2023-10-30T12:19:15.967 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_watch_notify on pid 92225' 2023-10-30T12:19:15.968 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92225 2023-10-30T12:19:15.968 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.968 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.968 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.969 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.969 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.969 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.969 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.969 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.969 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.970 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.970 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.970 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.970 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.970 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.971 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.971 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.971 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.971 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.971 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.972 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:15.972 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.972 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:15.972 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:15.972 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:15.972 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:15.973 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_watch_notify_pp 2023-10-30T12:19:15.973 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:15.973 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-10-30T12:19:15.973 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_watch_notify_pp' 2023-10-30T12:19:15.973 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.974 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:15.974 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.974 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:15.974 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:15.974 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.975 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:15.975 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.975 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:15.975 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.975 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:15.976 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.976 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:15.976 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:15.976 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:15.976 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:15.976 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92136/exe 2023-10-30T12:19:15.977 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:15.977 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:15.977 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.977 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:15.977 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:15.978 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:15.978 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_watch_notify_pp 2023-10-30T12:19:15.978 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:15.978 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:15.978 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:15.979 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:15.979 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:15.979 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:15.979 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:15.979 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:15.979 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:15.980 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:15.980 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:15.980 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:15.980 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:15.980 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:15.981 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.981 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:15.981 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.981 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:15.981 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:15.982 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.982 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:15.982 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.982 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:15.982 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.982 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:15.983 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.983 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:15.983 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.983 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:15.983 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.984 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_snapshots_pp 2023-10-30T12:19:15.984 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.984 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:15.984 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.984 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:15.984 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:15.985 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:15.985 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_watch_notify_pp 2023-10-30T12:19:15.985 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92249 2023-10-30T12:19:15.985 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_watch_notify_pp on pid 92249' 2023-10-30T12:19:15.986 INFO:tasks.workunit.client.0.smithi148.stdout:test api_watch_notify_pp on pid 92249 2023-10-30T12:19:15.993 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92249 2023-10-30T12:19:15.994 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:15.994 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:15.994 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-10-30T12:19:15.994 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_cmd 2023-10-30T12:19:15.994 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_cmd' 2023-10-30T12:19:15.995 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:15.995 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:15.995 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:15.995 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.995 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:15.996 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.996 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:15.996 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:15.996 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:15.996 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:15.998 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:15.998 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:15.998 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:15.998 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:15.999 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:15.999 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:15.999 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:15.999 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:15.999 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:15.999 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.000 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.000 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.000 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.000 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.000 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-10-30T12:19:16.001 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-10-30T12:19:16.001 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92192/exe 2023-10-30T12:19:16.001 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.001 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.001 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.001 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.002 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.002 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.002 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.002 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.002 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.003 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.003 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.003 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.003 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.003 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.004 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.004 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.004 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.004 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.004 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.004 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.005 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.005 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.005 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.005 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.005 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.006 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.006 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.006 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.006 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.006 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.007 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.007 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.007 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.007 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.007 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.008 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.008 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.008 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.008 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.008 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.008 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.009 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.009 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.009 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.009 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.009 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.010 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.010 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.010 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.010 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.010 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.011 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.011 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.011 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.011 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.011 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_cmd 2023-10-30T12:19:16.011 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.012 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:16.012 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.012 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-10-30T12:19:16.013 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [----------] Global test environment set-up. 2023-10-30T12:19:16.013 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [----------] 12 tests from AsioRados 2023-10-30T12:19:16.013 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.630+0000 7fd366e44780 10 monclient: get_monmap_and_config 2023-10-30T12:19:16.013 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.630+0000 7fd366e44780 10 monclient: build_initial_monmap 2023-10-30T12:19:16.013 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.630+0000 7fd366e44780 10 monclient: monmap: 2023-10-30T12:19:16.014 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: epoch 0 2023-10-30T12:19:16.014 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:16.014 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: last_changed 0.000000 2023-10-30T12:19:16.014 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: created 0.000000 2023-10-30T12:19:16.014 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: min_mon_release 0 (unknown) 2023-10-30T12:19:16.015 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: election_strategy: 1 2023-10-30T12:19:16.015 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:19:16.015 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:19:16.015 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:19:16.015 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.015 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.631+0000 7fd366e44780 1 Processor -- start 2023-10-30T12:19:16.016 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.631+0000 7fd366e44780 1 -- start start 2023-10-30T12:19:16.016 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.631+0000 7fd366e44780 10 monclient: init 2023-10-30T12:19:16.016 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 10 monclient: _reopen_session rank -1 2023-10-30T12:19:16.016 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:19:16.016 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x559edb4415a0 0x559edb5032f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:16.017 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 10 monclient(hunting): picked mon.noname-c con 0x559edb4415a0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:19:16.017 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559edb441ef0 0x559edb47d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:16.017 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 10 monclient(hunting): picked mon.noname-b con 0x559edb441ef0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:19:16.017 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 0x559edb47f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:16.018 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 10 monclient(hunting): picked mon.noname-a con 0x559edb50d0d0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:19:16.018 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:16.018 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x559edb383d20 con 0x559edb50d0d0 2023-10-30T12:19:16.018 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:16.018 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559edb381160 con 0x559edb441ef0 2023-10-30T12:19:16.019 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:16.019 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x559edb500cb0 con 0x559edb4415a0 2023-10-30T12:19:16.019 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 10 monclient(hunting): _renew_subs 2023-10-30T12:19:16.019 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.632+0000 7fd366e44780 10 monclient(hunting): authenticate will time out at 1647.837083s 2023-10-30T12:19:16.021 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.021 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.021 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.022 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.022 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.022 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.022 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.022 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.023 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.023 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.023 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.023 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.023 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.024 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.024 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.024 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.024 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.024 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_cmd 2023-10-30T12:19:16.024 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92315 2023-10-30T12:19:16.025 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_cmd on pid 92315' 2023-10-30T12:19:16.025 INFO:tasks.workunit.client.0.smithi148.stdout: test api_cmd on pid 92315 2023-10-30T12:19:16.029 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.029 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.029 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92315 2023-10-30T12:19:16.029 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.029 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.030 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.030 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:16.030 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:16.030 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.030 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_cmd_pp 2023-10-30T12:19:16.031 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_stat_pp 2023-10-30T12:19:16.031 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_cmd_pp' 2023-10-30T12:19:16.031 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-10-30T12:19:16.031 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.031 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.032 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.032 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.032 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.032 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.032 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.032 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.033 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.033 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.033 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_stat_pp: /' 2023-10-30T12:19:16.033 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.033 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.034 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.034 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.034 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.034 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.034 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.035 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.035 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.035 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.035 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.035 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:16.035 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-10-30T12:19:16.036 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_cmd_pp 2023-10-30T12:19:16.036 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_cmd_pp 2023-10-30T12:19:16.036 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92331 2023-10-30T12:19:16.036 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_cmd_pp on pid 92331' 2023-10-30T12:19:16.037 INFO:tasks.workunit.client.0.smithi148.stdout:test api_cmd_pp on pid 92331 2023-10-30T12:19:16.037 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92331 2023-10-30T12:19:16.038 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:16.038 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:16.038 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92177/exe 2023-10-30T12:19:16.038 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_service 2023-10-30T12:19:16.039 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.652+0000 7fd364a11640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 0x559edb47f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:16.039 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.652+0000 7fd364a11640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 0x559edb47f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60270/0 (socket says 172.21.15.148:60270) 2023-10-30T12:19:16.039 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.652+0000 7fd364a11640 1 -- 172.21.15.148:0/3670904732 learned_addr learned my addr 172.21.15.148:0/3670904732 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:19:16.039 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.652+0000 7fd364a11640 10 monclient(hunting): get_auth_request con 0x559edb50d0d0 auth_method 0 2023-10-30T12:19:16.039 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.652+0000 7fd364a11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:16.040 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.652+0000 7fd364a11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:16.040 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.652+0000 7fd364a11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:16.040 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.654+0000 7fd364a11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:16.040 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.654+0000 7fd364a11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:16.040 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.654+0000 7fd364a11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:16.041 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.661+0000 7fd363a0f640 1 --2- 172.21.15.148:0/3670904732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559edb441ef0 0x559edb47d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:16.041 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.661+0000 7fd363a0f640 10 monclient(hunting): get_auth_request con 0x559edb441ef0 auth_method 0 2023-10-30T12:19:16.041 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.661+0000 7fd363a0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:16.041 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.661+0000 7fd363a0f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:16.041 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.661+0000 7fd363a0f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:16.042 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd363a0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:16.042 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd363a0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:16.042 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd363a0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:16.042 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd364a11640 10 monclient(hunting): handle_auth_done global_id 4398 payload 274 2023-10-30T12:19:16.042 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd364a11640 1 -- 172.21.15.148:0/3670904732 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x559edb4415a0 msgr2=0x559edb5032f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:16.043 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd364a11640 1 --2- 172.21.15.148:0/3670904732 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x559edb4415a0 0x559edb5032f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:16.043 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd364a11640 1 -- 172.21.15.148:0/3670904732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559edb441ef0 msgr2=0x559edb47d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:16.054 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_service' 2023-10-30T12:19:16.054 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_service 2023-10-30T12:19:16.055 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.055 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.055 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-10-30T12:19:16.055 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.055 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.056 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.056 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.056 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.056 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.056 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.057 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.057 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.057 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.057 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.057 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.057 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.058 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.058 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.058 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.058 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.058 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.059 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.059 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.059 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.059 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.059 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.060 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.060 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.060 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.060 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.060 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.061 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.061 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.061 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.061 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.061 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.062 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.062 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.062 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.062 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.062 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.062 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.063 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.063 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.063 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.063 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.063 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.064 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.064 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.064 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.064 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.064 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.065 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_stat: /' 2023-10-30T12:19:16.065 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_stat.log 2023-10-30T12:19:16.065 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.065 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.065 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.066 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.066 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.066 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.066 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.066 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.067 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_stat 2023-10-30T12:19:16.067 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.067 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.067 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.067 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.068 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.068 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.068 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.068 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.068 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.069 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.069 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.069 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.069 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.069 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.070 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.070 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.070 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.070 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.070 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.070 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.071 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.071 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.071 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.071 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.071 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.072 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.072 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.072 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.072 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.072 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.073 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.073 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.073 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.073 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.073 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.074 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.074 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.074 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.074 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.074 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.075 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.075 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.075 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.075 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.075 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.075 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.076 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.076 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.076 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.076 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.076 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.077 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.077 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.077 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.077 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.077 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.078 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.078 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.078 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.078 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:16.078 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.079 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.079 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.079 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.079 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.079 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.079 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.080 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.080 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.080 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.080 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.080 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.081 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.081 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.081 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.081 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_service 2023-10-30T12:19:16.081 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.082 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.082 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.082 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92371 2023-10-30T12:19:16.082 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.082 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.083 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.083 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.083 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.083 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.083 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.083 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_service on pid 92371' 2023-10-30T12:19:16.084 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd364a11640 1 --2- 172.21.15.148:0/3670904732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55test api_service on pid 92371 2023-10-30T12:19:16.094 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92371 2023-10-30T12:19:16.094 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:16.094 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:16.095 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_service_pp 2023-10-30T12:19:16.095 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_service_pp' 2023-10-30T12:19:16.095 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.095 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.095 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.096 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.096 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.096 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.096 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.096 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.097 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.097 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.097 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.097 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.097 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.097 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.098 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.098 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.098 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.098 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.099 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.099 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.099 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.099 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.099 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.099 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.100 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.100 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.100 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.100 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.100 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.101 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.101 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.101 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.101 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.101 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.102 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.102 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.102 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.102 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.102 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.103 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.103 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.103 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.103 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.103 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.104 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.104 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.104 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.104 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.104 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.104 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.105 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-10-30T12:19:16.105 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:16.105 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92225/exe 2023-10-30T12:19:16.105 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.105 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.106 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.106 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_service_pp 2023-10-30T12:19:16.106 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.106 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.106 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.107 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.107 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.107 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.107 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.107 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.108 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.108 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.108 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.108 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.108 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.108 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.109 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.109 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.109 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.109 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.109 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.110 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.110 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.110 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.110 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.110 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.110 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.111 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.111 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.111 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.111 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.111 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.112 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.112 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.112 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.112 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.112 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.113 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.113 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.113 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.113 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.113 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.114 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.114 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.114 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.114 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.114 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.115 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.115 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.115 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.115 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.115 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.115 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.116 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.116 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.116 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.116 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.116 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.117 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.117 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.117 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.117 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.117 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.117 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.118 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.118 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.118 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.118 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.119 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.119 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.119 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_service_pp 2023-10-30T12:19:16.119 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.119 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.119 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92407 2023-10-30T12:19:16.120 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_service_pp on pid 92407' 2023-10-30T12:19:16.120 INFO:tasks.workunit.client.0.smithi148.stdout:test api_service_pp on pid 92407 2023-10-30T12:19:16.130 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92407 2023-10-30T12:19:16.130 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:16.130 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:16.131 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.131 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.131 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.131 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.131 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.131 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.132 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.132 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-10-30T12:19:16.132 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.132 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.132 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.133 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.133 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.133 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.133 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-10-30T12:19:16.133 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.134 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.134 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.134 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.134 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.134 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_c_write_operations 2023-10-30T12:19:16.135 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_c_write_operations' 2023-10-30T12:19:16.135 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_watch_notify: /' 2023-10-30T12:19:16.135 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.135 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.135 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.135 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.136 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.136 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.136 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.136 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.136 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_watch_notify 2023-10-30T12:19:16.137 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:16.137 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.137 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.137 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.137 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.137 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.138 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.138 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.138 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.138 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.138 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.139 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.139 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.139 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.139 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.139 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.141 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.141 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.141 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.141 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.141 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.142 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.142 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.142 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.142 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.142 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.142 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.143 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.143 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.143 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.143 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.143 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.144 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.144 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.144 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.144 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.144 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.144 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.145 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.145 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.145 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.145 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_c_write_operations 2023-10-30T12:19:16.145 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.146 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.146 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.146 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.146 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.146 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.146 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.147 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.147 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.147 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.147 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.147 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.148 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.148 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.148 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.148 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.148 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.148 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.149 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.149 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.149 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.149 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.149 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.150 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.150 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.150 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.150 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.150 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.151 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.151 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.151 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.151 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.151 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.151 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.152 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.152 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.152 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.152 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.152 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.153 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.153 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.153 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.153 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.153 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.154 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.154 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.154 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.154 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.154 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.155 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.155 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.155 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.155 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.155 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.155 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.156 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.156 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.156 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_c_write_operations 2023-10-30T12:19:16.156 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.156 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92427 2023-10-30T12:19:16.157 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_c_write_operations on pid 92427' 2023-10-30T12:19:16.157 INFO:tasks.workunit.client.0.smithi148.stdout:test api_c_write_operations on pid 92427 2023-10-30T12:19:16.166 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92427 2023-10-30T12:19:16.167 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:16.167 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:16.167 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92315/exe 2023-10-30T12:19:16.167 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.168 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.168 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.168 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92249/exe 2023-10-30T12:19:16.168 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.168 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.168 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.169 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.169 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.169 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.169 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.169 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.170 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.170 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.170 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.170 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.170 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.171 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.171 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.171 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.171 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.171 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.171 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-10-30T12:19:16.172 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.172 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.172 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.172 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.172 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.173 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.173 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.173 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.173 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.173 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.174 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.174 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.174 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.174 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.174 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.175 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.175 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.175 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_c_read_operations 2023-10-30T12:19:16.175 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.175 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.176 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.176 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.176 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.176 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_c_read_operations' 2023-10-30T12:19:16.176 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.177 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.177 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.177 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.177 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.177 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.177 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.180 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_cmd 2023-10-30T12:19:16.180 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.180 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.180 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.180 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.180 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_cmd: /' 2023-10-30T12:19:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_c_read_operations 2023-10-30T12:19:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-10-30T12:19:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.183 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.183 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.183 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.183 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.183 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.184 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.184 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.184 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_cmd.log 2023-10-30T12:19:16.184 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.184 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.184 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.185 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.185 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.185 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.185 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.185 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.185 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.186 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.186 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_c_read_operations 2023-10-30T12:19:16.186 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92455 2023-10-30T12:19:16.186 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_c_read_operations on pid 92455' 2023-10-30T12:19:16.187 INFO:tasks.workunit.client.0.smithi148.stdout:test api_c_read_operations on pid 92455 2023-10-30T12:19:16.198 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92455 2023-10-30T12:19:16.199 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_watch_notify_pp 2023-10-30T12:19:16.199 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:16.199 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:16.199 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.199 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s api_cls_remote_reads 2023-10-30T12:19:16.200 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-10-30T12:19:16.200 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.200 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.200 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.200 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92331/exe 2023-10-30T12:19:16.201 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.201 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.201 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.201 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' api_cls_remote_reads' 2023-10-30T12:19:16.201 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.202 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.202 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.202 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.202 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-10-30T12:19:16.202 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.202 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.203 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.203 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.203 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.203 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.203 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.204 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.204 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo api_cls_remote_reads 2023-10-30T12:19:16.204 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.204 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.204 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.204 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.205 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.205 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.205 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.205 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.205 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.206 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.206 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.206 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.206 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.206 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.207 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.207 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.207 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.207 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.207 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.207 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.208 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.208 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.208 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.208 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.208 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.209 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.209 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.209 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.209 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.209 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.209 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.210 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.210 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.210 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.210 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.210 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.211 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-10-30T12:19:16.211 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_cmd_pp 2023-10-30T12:19:16.211 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.211 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.211 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.211 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.212 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.212 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.212 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.212 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.212 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.213 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.213 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.213 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.213 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.213 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.213 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.214 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.214 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.214 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-10-30T12:19:16.214 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.214 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.215 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.215 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.215 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.215 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.215 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.216 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.216 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.216 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.216 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.216 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.216 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.217 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.217 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.217 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.217 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.217 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.218 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.218 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.218 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.218 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.218 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.218 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.219 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.219 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.219 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.219 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:16.219 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.220 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.220 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.220 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.220 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.220 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.220 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.221 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.221 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.221 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.221 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.221 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.222 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.222 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.222 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.222 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.222 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.223 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.223 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.223 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.223 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.223 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.224 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.224 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.224 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.225 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.225 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.226 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.226 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.226 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.227 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.227 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.227 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.227 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.227 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.228 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.228 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.228 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.228 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.229 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.229 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.229 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=api_cls_remote_reads 2023-10-30T12:19:16.229 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92478 2023-10-30T12:19:16.230 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test api_cls_remote_reads on pid 92478' 2023-10-30T12:19:16.230 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-10-30T12:19:16.231 INFO:tasks.workunit.client.0.smithi148.stdout:test api_cls_remote_reads on pid 92478 2023-10-30T12:19:16.242 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92478 2023-10-30T12:19:16.242 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:16.242 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:16.242 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.243 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.243 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.243 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.244 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.244 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.244 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.244 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.244 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.245 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.245 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.245 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.245 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.245 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.246 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.246 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.246 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.246 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s list_parallel 2023-10-30T12:19:16.246 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' list_parallel' 2023-10-30T12:19:16.247 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.247 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92371/exe 2023-10-30T12:19:16.247 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.247 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.247 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.248 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.248 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.248 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.248 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.248 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.249 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.249 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.249 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.249 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.250 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.250 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.250 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.250 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.250 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.251 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.251 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.251 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.251 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.251 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.252 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.252 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.252 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.252 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.252 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.253 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.253 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.253 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.253 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.257 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.257 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.257 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.257 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.258 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.258 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_service.log 2023-10-30T12:19:16.258 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.258 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_service 2023-10-30T12:19:16.258 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.259 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.259 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.259 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.259 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.260 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.260 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.260 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.260 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.261 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo list_parallel 2023-10-30T12:19:16.261 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.261 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.261 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.261 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.262 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.262 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.263 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:16.263 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.263 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.264 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.264 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.264 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.264 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.264 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.265 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.265 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.265 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.265 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.265 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.266 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.266 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.266 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.267 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.267 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.267 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.268 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.268 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.269 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.269 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.269 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.269 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.269 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.270 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.270 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.270 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.270 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.270 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.271 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.271 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.271 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.271 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.271 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.272 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.272 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.272 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.272 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.272 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.272 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.273 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.273 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.273 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=list_parallel 2023-10-30T12:19:16.273 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92517 2023-10-30T12:19:16.273 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test list_parallel on pid 92517' 2023-10-30T12:19:16.274 INFO:tasks.workunit.client.0.smithi148.stdout:test list_parallel on pid 92517 2023-10-30T12:19:16.289 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92517 2023-10-30T12:19:16.289 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:16.289 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:16.290 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.290 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.290 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.290 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.291 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.291 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.291 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.291 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.291 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.292 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s open_pools_parallel 2023-10-30T12:19:16.292 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' open_pools_parallel' 2023-10-30T12:19:16.292 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.292 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.293 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.293 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.293 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.293 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.293 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.294 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92407/exe 2023-10-30T12:19:16.294 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.294 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.295 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.295 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.295 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.295 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.295 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.296 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.296 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.296 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.296 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.297 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.297 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.297 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.297 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.298 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.298 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.298 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.298 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.298 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.299 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.299 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.299 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.300 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.300 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.300 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.300 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.301 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.301 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.301 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.302 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.302 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_service_pp: /' 2023-10-30T12:19:16.302 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.302 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.302 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_service_pp 2023-10-30T12:19:16.303 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.303 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.303 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.303 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.303 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.304 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.304 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.304 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-10-30T12:19:16.304 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.304 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.304 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.305 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.305 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.305 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.305 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.306 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.306 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.306 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.306 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.307 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-10-30T12:19:16.307 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.307 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.307 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.307 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.307 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.308 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.308 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.308 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.308 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.308 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.309 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_service: /' 2023-10-30T12:19:16.309 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.309 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.309 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.309 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.311 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.311 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.311 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.311 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.311 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.312 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.312 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.312 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.312 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.312 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.312 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.313 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.313 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.313 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.313 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.313 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.314 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.314 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.314 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92478/exe 2023-10-30T12:19:16.314 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.314 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.315 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.315 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.315 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.315 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.315 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.316 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.316 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.316 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.316 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.316 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.317 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.317 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo open_pools_parallel 2023-10-30T12:19:16.317 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.317 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.317 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.318 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.318 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.318 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.318 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.318 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.319 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:16.319 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.319 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.319 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.319 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.320 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.320 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.320 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.320 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.321 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.321 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.321 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.321 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.322 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.322 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.322 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.322 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.323 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.323 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.323 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.323 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.324 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.324 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.324 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.324 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.325 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.325 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.325 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.325 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.326 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.326 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.326 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.327 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.327 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.327 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.327 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.327 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.328 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.328 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.328 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.328 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92427/exe 2023-10-30T12:19:16.329 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.329 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_cls_remote_reads 2023-10-30T12:19:16.329 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-10-30T12:19:16.329 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-10-30T12:19:16.330 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.330 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.330 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.330 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.331 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.331 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.331 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.331 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.332 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.332 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.332 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_c_write_operations 2023-10-30T12:19:16.332 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=open_pools_parallel 2023-10-30T12:19:16.333 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.333 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92592 2023-10-30T12:19:16.333 INFO:tasks.workunit.client.0.smithi148.stdout:test open_pools_parallel on pid 92592 2023-10-30T12:19:16.342 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test open_pools_parallel on pid 92592' 2023-10-30T12:19:16.342 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92592 2023-10-30T12:19:16.342 INFO:tasks.workunit.client.0.smithi148.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:19:16.343 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:16.343 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-10-30T12:19:16.343 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.343 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.343 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.344 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.344 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.344 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.344 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92455/exe 2023-10-30T12:19:16.344 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.345 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.345 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.345 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.345 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.345 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.346 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.346 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.346 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.346 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.346 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.347 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.347 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.347 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.347 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.347 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.347 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.348 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.348 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.348 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-10-30T12:19:16.348 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.348 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.349 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.349 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.349 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.349 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.349 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.350 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.350 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.350 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.350 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.350 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.351 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.351 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.351 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.351 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.351 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.352 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-10-30T12:19:16.352 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.352 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.352 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.353 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.353 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.353 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.353 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.353 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.354 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.354 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.354 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.354 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.354 INFO:tasks.workunit.client.0.smithi148.stderr:++ printf %25s delete_pools_parallel 2023-10-30T12:19:16.355 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=' delete_pools_parallel' 2023-10-30T12:19:16.355 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_api_c_read_operations 2023-10-30T12:19:16.355 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.355 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.355 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.356 INFO:tasks.workunit.client.0.smithi148.stderr:++ awk '{print $1}' 2023-10-30T12:19:16.356 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.356 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.356 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.356 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.357 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.357 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.357 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.357 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.357 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.358 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.358 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.358 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.358 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.358 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.358 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.359 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.359 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.359 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-10-30T12:19:16.359 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-10-30T12:19:16.359 INFO:tasks.workunit.client.0.smithi148.stderr:++ echo delete_pools_parallel 2023-10-30T12:19:16.360 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.360 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.360 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.360 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.361 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.361 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.361 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.361 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.361 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.362 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.362 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.362 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.365 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.366 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.366 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.366 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.366 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.367 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.367 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.367 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.367 INFO:tasks.workunit.client.0.smithi148.stderr:+ ff=delete_pools_parallel 2023-10-30T12:19:16.367 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92632 2023-10-30T12:19:16.368 INFO:tasks.workunit.client.0.smithi148.stdout:test delete_pools_parallel on pid 92632 2023-10-30T12:19:16.390 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 'test delete_pools_parallel on pid 92632' 2023-10-30T12:19:16.390 INFO:tasks.workunit.client.0.smithi148.stderr:+ pids[$f]=92632 2023-10-30T12:19:16.390 INFO:tasks.workunit.client.0.smithi148.stderr:+ ret=0 2023-10-30T12:19:16.391 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:19:16.391 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:19:16.391 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92048 2023-10-30T12:19:16.391 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92048 2023-10-30T12:19:16.391 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.391 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.392 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.392 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.392 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.392 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.392 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.393 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.393 INFO:tasks.workunit.client.0.smithi148.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-10-30T12:19:16.393 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.393 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:19:16.393 INFO:tasks.workunit.client.0.smithi148.stderr:+ . /etc/bashrc 2023-10-30T12:19:16.394 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -z '' ']' 2023-10-30T12:19:16.394 INFO:tasks.workunit.client.0.smithi148.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:19:16.394 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.394 INFO:tasks.workunit.client.0.smithi148.stderr:++ shopt -q login_shell 2023-10-30T12:19:16.394 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.395 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.395 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.395 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.395 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.395 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.396 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.396 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.396 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.396 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.397 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.397 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.397 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.397 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.397 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.398 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.398 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.398 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.398 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.399 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.399 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.399 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.399 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.399 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.400 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.400 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.400 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.400 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.401 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.401 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.401 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.401 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.401 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.402 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.402 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.402 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.402 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.403 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.403 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.403 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.403 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.403 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.404 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.404 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.404 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.404 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.405 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.405 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.405 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.405 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.406 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.406 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.406 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.406 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.407 INFO:tasks.workunit.client.0.smithi148.stderr:+++ umask 2023-10-30T12:19:16.407 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:19:16.407 INFO:tasks.workunit.client.0.smithi148.stderr:++ SHELL=/bin/bash 2023-10-30T12:19:16.407 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.408 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:19:16.408 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.408 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:19:16.408 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92592/exe 2023-10-30T12:19:16.409 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:19:16.409 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.409 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:19:16.409 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.409 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:19:16.410 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.410 INFO:tasks.workunit.client.0.smithi148.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:19:16.410 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.410 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:19:16.410 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.411 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:19:16.411 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.411 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.411 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.411 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.411 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.412 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.412 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.412 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.412 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.412 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.413 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.413 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.413 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.413 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.413 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.414 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.414 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.414 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.414 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.414 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.415 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.415 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.415 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.415 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.415 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.416 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.416 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.416 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.416 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.416 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.417 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:19:16.417 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:19:16.417 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:19:16.417 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.417 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:19:16.418 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.418 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:19:16.418 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:19:16.418 INFO:tasks.workunit.client.0.smithi148.stderr:+++ return 2023-10-30T12:19:16.418 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.418 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:19:16.419 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.419 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:19:16.419 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export S_COLORS=auto 2023-10-30T12:19:16.419 INFO:tasks.workunit.client.0.smithi148.stderr:+++ S_COLORS=auto 2023-10-30T12:19:16.419 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.420 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:19:16.420 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.420 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:19:16.420 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.420 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.421 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.421 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.421 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.421 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.421 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-10-30T12:19:16.421 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.422 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:19:16.422 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:19:16.422 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:19:16.422 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.422 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:19:16.423 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.423 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:19:16.423 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:19:16.423 INFO:tasks.workunit.client.0.smithi148.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:19:16.423 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_open_pools_parallel 2023-10-30T12:19:16.423 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.424 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.424 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.424 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.424 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.424 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.425 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.425 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.425 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.425 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.425 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-10-30T12:19:16.425 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.426 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.426 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.426 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.426 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.426 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.427 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.427 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.427 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.427 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.427 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.428 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92517/exe 2023-10-30T12:19:16.428 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:19:16.428 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:19:16.428 INFO:tasks.workunit.client.0.smithi148.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:19:16.428 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.428 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.429 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:19:16.429 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.429 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:19:16.430 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:19:16.430 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.430 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:19:16.430 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.430 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:19:16.431 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.431 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:19:16.431 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.431 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:19:16.431 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup 2023-10-30T12:19:16.432 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.432 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.432 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.432 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.432 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/locale 2023-10-30T12:19:16.432 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.433 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.433 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.433 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.433 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.433 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.434 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_list_parallel 2023-10-30T12:19:16.434 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_list_parallel.log 2023-10-30T12:19:16.434 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.434 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:19:16.435 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.435 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:19:16.435 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:19:16.435 INFO:tasks.workunit.client.0.smithi148.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:19:16.435 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ list_parallel: /' 2023-10-30T12:19:16.436 INFO:tasks.workunit.client.0.smithi148.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:19:16.436 INFO:tasks.workunit.client.0.smithi148.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:19:16.436 INFO:tasks.workunit.client.0.smithi148.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:19:16.436 INFO:tasks.workunit.client.0.smithi148.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:19:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:19:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:19:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:+++ unset LANG_backup config 2023-10-30T12:19:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n '' ']' 2023-10-30T12:19:16.438 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:19:16.438 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:19:16.438 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.438 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:19:16.439 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.439 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:19:16.439 INFO:tasks.workunit.client.0.smithi148.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:19:16.439 INFO:tasks.workunit.client.0.smithi148.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:19:16.439 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:19:16.440 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' '' ']' 2023-10-30T12:19:16.440 INFO:tasks.workunit.client.0.smithi148.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:19:16.440 INFO:tasks.workunit.client.0.smithi148.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:19:16.440 INFO:tasks.workunit.client.0.smithi148.stderr:+++++ readlink /proc/92632/exe 2023-10-30T12:19:16.441 INFO:tasks.workunit.client.0.smithi148.stderr:++++ basename /usr/bin/bash 2023-10-30T12:19:16.441 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_declare='declare -f' 2023-10-30T12:19:16.441 INFO:tasks.workunit.client.0.smithi148.stderr:+++ which_opt=-f 2023-10-30T12:19:16.441 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export which_declare 2023-10-30T12:19:16.441 INFO:tasks.workunit.client.0.smithi148.stderr:+++ export -f which 2023-10-30T12:19:16.442 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset i 2023-10-30T12:19:16.442 INFO:tasks.workunit.client.0.smithi148.stderr:++ unset -f pathmunge 2023-10-30T12:19:16.442 INFO:tasks.workunit.client.0.smithi148.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:19:16.443 INFO:tasks.workunit.client.0.smithi148.stderr:+ export PATH 2023-10-30T12:19:16.443 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:19:16.443 INFO:tasks.workunit.client.0.smithi148.stderr:+ unset rc 2023-10-30T12:19:16.444 INFO:tasks.workunit.client.0.smithi148.stderr:+ ceph_test_rados_delete_pools_parallel 2023-10-30T12:19:16.444 INFO:tasks.workunit.client.0.smithi148.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-10-30T12:19:16.444 INFO:tasks.workunit.client.0.smithi148.stdout:9edb441ef0 0x559edb47d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:16.445 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd364a11640 10 monclient: _finish_hunting 0 2023-10-30T12:19:16.445 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd364a11640 1 monclient: found mon.noname-a 2023-10-30T12:19:16.445 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd364a11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:16.445 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd364a11640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:19:16.445 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.664+0000 7fd364a11640 1 -- 172.21.15.148:0/3670904732 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559edb47dcc0 con 0x559edb50d0d0 2023-10-30T12:19:16.446 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.668+0000 7fd364a11640 1 --2- 172.21.15.148:0/3670904732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 0x559edb47f0e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00a7a0 tx=0x7fd35c00ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f62ca920051ff7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:16.446 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 1 -- 172.21.15.148:0/3670904732 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00d9c0 con 0x559edb50d0d0 2023-10-30T12:19:16.446 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:16.446 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:16.446 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 10 monclient: dump: 2023-10-30T12:19:16.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: epoch 1 2023-10-30T12:19:16.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:19:16.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:19:16.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:19:16.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:19:16.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: election_strategy: 3 2023-10-30T12:19:16.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:19:16.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:19:16.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:19:16.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 10 monclient: _finish_auth 0 2023-10-30T12:19:16.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 10 monclient: _check_auth_tickets 2023-10-30T12:19:16.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:16.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd366e44780 5 monclient: authenticate success, global_id 4398 2023-10-30T12:19:16.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 1 -- 172.21.15.148:0/3670904732 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c00c040 con 0x559edb50d0d0 2023-10-30T12:19:16.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd366e44780 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:16.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:16.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 1 -- 172.21.15.148:0/3670904732 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00dd30 con 0x559edb50d0d0 2023-10-30T12:19:16.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:16.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:16.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd362a0d640 10 monclient: dump: 2023-10-30T12:19:16.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: epoch 1 2023-10-30T12:19:16.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:19:16.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:19:16.451 INFO:tasks.workunit.client.0.smithi148.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-10-30T12:19:16.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:19:16.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:19:16.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: election_strategy: 3 2023-10-30T12:19:16.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:19:16.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:19:16.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:19:16.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd366e44780 10 monclient: get_monmap_and_config success 2023-10-30T12:19:16.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.830+0000 7fd366e44780 4 set_mon_vals no callback set 2023-10-30T12:19:16.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.831+0000 7fd366e44780 10 monclient: shutdown 2023-10-30T12:19:16.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.831+0000 7fd366e44780 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:16.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.831+0000 7fd366e44780 1 -- 172.21.15.148:0/3670904732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 msgr2=0x559edb47f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:16.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.831+0000 7fd366e44780 1 --2- 172.21.15.148:0/3670904732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 0x559edb47f0e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00a7a0 tx=0x7fd35c00ac60 comp rx=0 tx=0).stop 2023-10-30T12:19:16.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 1 -- 172.21.15.148:0/3670904732 shutdown_connections 2023-10-30T12:19:16.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 1 --2- 172.21.15.148:0/3670904732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 0x559edb47f0e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:16.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 1 --2- 172.21.15.148:0/3670904732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559edb441ef0 0x559edb47d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:16.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 1 --2- 172.21.15.148:0/3670904732 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x559edb4415a0 0x559edb5032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:16.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 1 -- 172.21.15.148:0/3670904732 >> 172.21.15.148:0/3670904732 conn(0x559edb43b750 msgr2=0x559edb43bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:16.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 1 -- 172.21.15.148:0/3670904732 shutdown_connections 2023-10-30T12:19:16.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 1 -- 172.21.15.148:0/3670904732 wait complete. 2023-10-30T12:19:16.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 10 monclient: build_initial_monmap 2023-10-30T12:19:16.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 10 monclient: monmap: 2023-10-30T12:19:16.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: epoch 0 2023-10-30T12:19:16.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:16.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: last_changed 0.000000 2023-10-30T12:19:16.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: created 0.000000 2023-10-30T12:19:16.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: min_mon_release 0 (unknown) 2023-10-30T12:19:16.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: election_strategy: 1 2023-10-30T12:19:16.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:19:16.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:19:16.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:19:16.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 1 librados: starting msgr at 2023-10-30T12:19:16.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 1 librados: starting objecter 2023-10-30T12:19:16.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.832+0000 7fd366e44780 1 Processor -- start 2023-10-30T12:19:16.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 1 -- start start 2023-10-30T12:19:16.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 1 librados: setting wanted keys 2023-10-30T12:19:16.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 1 librados: calling monclient init 2023-10-30T12:19:16.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient: init 2023-10-30T12:19:16.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient: _reopen_session rank -1 2023-10-30T12:19:16.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:19:16.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x559edb4415a0 0x559edb505440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:16.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient(hunting): picked mon.noname-c con 0x559edb4415a0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:19:16.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559edb441ef0 0x559edb507700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:16.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient(hunting): picked mon.noname-b con 0x559edb441ef0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:19:16.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 0x559edb507c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:16.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient(hunting): picked mon.noname-a con 0x559edb50d0d0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:19:16.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:16.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x559edb479a30 con 0x559edb50d0d0 2023-10-30T12:19:16.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:16.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559edb5ac250 con 0x559edb441ef0 2023-10-30T12:19:16.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:16.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x559edb475ee0 con 0x559edb4415a0 2023-10-30T12:19:16.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient(hunting): _renew_subs 2023-10-30T12:19:16.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.833+0000 7fd366e44780 10 monclient(hunting): authenticate will time out at 1648.038812s 2023-10-30T12:19:16.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.887+0000 7fd363a0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559edb441ef0 0x559edb507700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:16.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.887+0000 7fd363a0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559edb441ef0 0x559edb507700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:36870/0 (socket says 172.21.15.148:36870) 2023-10-30T12:19:16.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.887+0000 7fd363a0f640 1 -- 172.21.15.148:0/2607214554 learned_addr learned my addr 172.21.15.148:0/2607214554 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:19:16.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.887+0000 7fd363a0f640 10 monclient(hunting): get_auth_request con 0x559edb441ef0 auth_method 0 2023-10-30T12:19:16.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.887+0000 7fd363a0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:16.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.887+0000 7fd363a0f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:16.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.887+0000 7fd363a0f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:16.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.888+0000 7fd364210640 1 --2- 172.21.15.148:0/2607214554 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x559edb4415a0 0x559edb505440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:16.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.895+0000 7fd363a0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:16.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.895+0000 7fd363a0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:16.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.895+0000 7fd363a0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:16.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.916+0000 7fd364210640 10 monclient(hunting): get_auth_request con 0x559edb4415a0 auth_method 0 2023-10-30T12:19:16.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.916+0000 7fd364210640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:16.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.916+0000 7fd364210640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:16.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.916+0000 7fd364210640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:16.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.918+0000 7fd364210640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:16.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.918+0000 7fd364210640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:16.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.918+0000 7fd364210640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:16.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.925+0000 7fd364a11640 1 --2- 172.21.15.148:0/2607214554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 0x559edb507c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:16.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.926+0000 7fd364a11640 10 monclient(hunting): get_auth_request con 0x559edb50d0d0 auth_method 0 2023-10-30T12:19:16.468 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.926+0000 7fd364a11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:16.468 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.926+0000 7fd364a11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:16.468 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.926+0000 7fd364a11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:16.468 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.927+0000 7fd364210640 10 monclient(hunting): handle_auth_done global_id 4379 payload 824 2023-10-30T12:19:16.469 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.927+0000 7fd364210640 1 -- 172.21.15.148:0/2607214554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559edb441ef0 msgr2=0x559edb507700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:16.469 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.927+0000 7fd364210640 1 --2- 172.21.15.148:0/2607214554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559edb441ef0 0x559edb507700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:16.469 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.927+0000 7fd364210640 1 -- 172.21.15.148:0/2607214554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 msgr2=0x559edb507c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:16.469 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.927+0000 7fd364210640 1 --2- 172.21.15.148:0/2607214554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559edb50d0d0 0x559edb507c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:16.469 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.927+0000 7fd364210640 10 monclient: _finish_hunting 0 2023-10-30T12:19:16.470 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.927+0000 7fd364210640 1 monclient: found mon.noname-c 2023-10-30T12:19:16.470 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.927+0000 7fd364210640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:16.470 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.927+0000 7fd364210640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:19:16.471 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.927+0000 7fd364210640 1 -- 172.21.15.148:0/2607214554 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559edb509540 con 0x559edb4415a0 2023-10-30T12:19:16.471 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.938+0000 7fd364210640 1 --2- 172.21.15.148:0/2607214554 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x559edb4415a0 0x559edb505440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd35400aec0 tx=0x7fd3540086e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:16.471 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.938+0000 7fd361a0b640 10 client.?.objecter ms_handle_connect 0x559edb4415a0 2023-10-30T12:19:16.471 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.938+0000 7fd361a0b640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:16.471 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35400b810 con 0x559edb4415a0 2023-10-30T12:19:16.472 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:16.472 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:19:16.472 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 10 monclient: dump: 2023-10-30T12:19:16.473 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: epoch 1 2023-10-30T12:19:16.473 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:19:16.473 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:19:16.473 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:19:16.473 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:19:16.474 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: election_strategy: 3 2023-10-30T12:19:16.474 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:19:16.474 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:19:16.474 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:19:16.475 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.475 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 10 monclient: _finish_auth 0 2023-10-30T12:19:16.475 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 10 monclient: _check_auth_tickets 2023-10-30T12:19:16.475 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:16.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35400b980 con 0x559edb4415a0 2023-10-30T12:19:16.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:16.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35400bd40 con 0x559edb4415a0 2023-10-30T12:19:16.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:16.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:16.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd361a0b640 10 monclient: dump: 2023-10-30T12:19:16.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: epoch 1 2023-10-30T12:19:16.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:19:16.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:19:16.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:19:16.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:19:16.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: election_strategy: 3 2023-10-30T12:19:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:19:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:19:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:19:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd362a0d640 4 set_mon_vals no callback set 2023-10-30T12:19:16.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd366e44780 5 monclient: authenticate success, global_id 4379 2023-10-30T12:19:16.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd366e44780 10 monclient: _renew_subs 2023-10-30T12:19:16.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd366e44780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:19:16.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559edb43bb70 con 0x559edb4415a0 2023-10-30T12:19:16.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd366e44780 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:16.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd366e44780 10 monclient: _renew_subs 2023-10-30T12:19:16.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd366e44780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:19:16.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559edb479db0 con 0x559edb4415a0 2023-10-30T12:19:16.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd366e44780 1 librados: init done 2023-10-30T12:19:16.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:15.997+0000 7fd366e44780 10 librados: wait_for_osdmap waiting 2023-10-30T12:19:16.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.383+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fd354011070 con 0x559edb4415a0 2023-10-30T12:19:16.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.384+0000 7fd361a0b640 1 --2- 172.21.15.148:0/2607214554 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd35006a9b0 0x7fd35006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:16.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.384+0000 7fd363a0f640 1 --2- 172.21.15.148:0/2607214554 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd35006a9b0 0x7fd35006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:16.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.384+0000 7fd363a0f640 10 monclient: get_auth_request con 0x7fd35006a9b0 auth_method 0 2023-10-30T12:19:16.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.384+0000 7fd363a0f640 1 --2- 172.21.15.148:0/2607214554 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd35006a9b0 0x7fd35006ce70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x559edb440f10 tx=0x7fd358052000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:16.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.384+0000 7fd361a0b640 10 client.4379.objecter ms_handle_connect 0x7fd35006a9b0 2023-10-30T12:19:16.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.387+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fd354056e40 con 0x559edb4415a0 2023-10-30T12:19:16.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.387+0000 7fd361a0b640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_map(24..24 src has 1..24) v4 2023-10-30T12:19:16.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.387+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map got epochs [24,24] > 0 2023-10-30T12:19:16.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.387+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding full epoch 24 2023-10-30T12:19:16.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.388+0000 7fd361a0b640 20 client.4379.objecter dump_active .. 0 homeless 2023-10-30T12:19:16.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.388+0000 7fd366e44780 10 librados: wait_for_osdmap done waiting 2023-10-30T12:19:16.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.388+0000 7fd366e44780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:19:18.704 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.388+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x559edb3eee50 con 0x559edb4415a0 2023-10-30T12:19:18.705 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.420+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x559edb475ee0 con 0x559edb4415a0 2023-10-30T12:19:18.705 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.420+0000 7fd361a0b640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-10-30T12:19:18.705 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.420+0000 7fd36220c640 10 client.4379.objecter _get_latest_version latest 24, have it 2023-10-30T12:19:18.705 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.420+0000 7fd366e44780 10 client.4379.objecter create_pool name=ceph_test_rados_api_asio 2023-10-30T12:19:18.705 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.420+0000 7fd366e44780 10 client.4379.objecter pool_op_submit 1 2023-10-30T12:19:18.706 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.420+0000 7fd366e44780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:19:18.706 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:16.420+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x559edb4000d0 con 0x559edb4415a0 2023-10-30T12:19:18.706 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.675+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== mon.2 v2:172.21.15.148:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v26) v1 ==== 43+0+0 (secure 0 0 0) 0x7fd354019020 con 0x559edb4415a0 2023-10-30T12:19:18.706 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.675+0000 7fd361a0b640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 pool_op_reply(tid 1 (0) Success v26) v1 2023-10-30T12:19:18.706 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.675+0000 7fd361a0b640 10 client.4379.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v26) v1 2023-10-30T12:19:18.707 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.675+0000 7fd361a0b640 10 client.4379.objecter have request 1 at 0x559edb43bd70 Op: create 2023-10-30T12:19:18.707 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.675+0000 7fd361a0b640 20 client.4379.objecter waiting for client to reach epoch 26 before calling back 2023-10-30T12:19:18.707 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.675+0000 7fd361a0b640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:18.707 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.675+0000 7fd361a0b640 10 monclient: _renew_subs 2023-10-30T12:19:18.707 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.675+0000 7fd361a0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:19:18.708 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.675+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=25}) v3 -- 0x7fd354053210 con 0x559edb4415a0 2023-10-30T12:19:18.708 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.675+0000 7fd361a0b640 10 client.4379.objecter done 2023-10-30T12:19:18.708 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.702+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== mon.2 v2:172.21.15.148:3301/0 8 ==== osd_map(25..26 src has 1..26) v4 ==== 14523+0+0 (secure 0 0 0) 0x7fd354056740 con 0x559edb4415a0 2023-10-30T12:19:18.708 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.702+0000 7fd361a0b640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_map(25..26 src has 1..26) v4 2023-10-30T12:19:18.708 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.702+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map got epochs [25,26] > 24 2023-10-30T12:19:18.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.702+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 25 2023-10-30T12:19:18.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.702+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 26 2023-10-30T12:19:18.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd361a0b640 20 client.4379.objecter dump_active .. 0 homeless 2023-10-30T12:19:18.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 10 librados: set snap read head -> 1 2023-10-30T12:19:18.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 10 librados: writefull oid=exist nspace= 2023-10-30T12:19:18.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5a4f50 2023-10-30T12:19:20.403 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:02:35.477 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:20.455 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:02:35.529 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:20.484 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:02:35.531 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:20.521 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:35.595 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:20.523 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:02:35.570 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:20.531 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:35.605 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:20.574 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:02:35.621 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:20.591 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:35.665 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:20.675 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:02:35.748 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:21.835 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 26 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:21.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:21.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:21.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 1 --2- 172.21.15.148:0/2607214554 >> v2:172.21.15.148:6805/2395694124 conn(0x559edb509a60 0x559edb5a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:21.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:21.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [writefull 0~5 in=5b] tid 2 osd.0 2023-10-30T12:19:21.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:21.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 2 2023-10-30T12:19:21.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 15 client.4379.objecter _send_op 2 to 20.1 on osd.0 2023-10-30T12:19:21.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:2 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) v8 -- 0x559edb5bffc0 con 0x559edb509a60 2023-10-30T12:19:21.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:21.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:21.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.703+0000 7fd364a11640 1 --2- 172.21.15.148:0/2607214554 >> v2:172.21.15.148:6805/2395694124 conn(0x559edb509a60 0x559edb5a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:21.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.704+0000 7fd364a11640 10 monclient: get_auth_request con 0x559edb509a60 auth_method 0 2023-10-30T12:19:21.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.707+0000 7fd361a0b640 10 client.4379.objecter ms_handle_connect 0x559edb509a60 2023-10-30T12:19:21.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.707+0000 7fd364a11640 1 --2- 172.21.15.148:0/2607214554 >> v2:172.21.15.148:6805/2395694124 conn(0x559edb509a60 0x559edb5a5900 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:21.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.833+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:21.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.833+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:21.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.833+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:21.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:18.833+0000 7fd360a09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:21.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:19.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:21.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:19.834+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:21.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:19.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:21.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:20.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:21.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:20.834+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:21.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:20.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:21.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:20.998+0000 7fd366474640 10 client.4379.objecter tick 2023-10-30T12:19:21.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:21.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:26.918 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:21.834+0000 7fd api_service: [==========] Running 4 tests from 1 test suite. 2023-10-30T12:19:26.919 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [----------] Global test environment set-up. 2023-10-30T12:19:26.919 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [----------] 4 tests from LibRadosService 2023-10-30T12:19:26.919 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-10-30T12:19:26.919 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5278 ms) 2023-10-30T12:19:26.920 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-10-30T12:19:26.920 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [ OK ] LibRadosService.RegisterLate (391 ms) 2023-10-30T12:19:26.920 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-10-30T12:19:26.920 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: cluster: 2023-10-30T12:19:26.920 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: id: e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:19:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: health: HEALTH_WARN 2023-10-30T12:19:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 10 pool(s) do not have an application enabled 2023-10-30T12:19:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: services: 2023-10-30T12:19:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-10-30T12:19:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: mgr: y(active, since 2m), standbys: x 2023-10-30T12:19:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: osd: 8 osds: 8 up (since 90s), 8 in (since 2m) 2023-10-30T12:19:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-10-30T12:19:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: data: 2023-10-30T12:19:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: pools: 26 pools, 777 pgs 2023-10-30T12:19:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: objects: 2 objects, 449 KiB 2023-10-30T12:19:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-10-30T12:19:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: pgs: 81.725% pgs unknown 2023-10-30T12:19:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 17.117% pgs not active 2023-10-30T12:19:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 635 unknown 2023-10-30T12:19:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 133 creating+peering 2023-10-30T12:19:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 9 active+clean 2023-10-30T12:19:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: cluster: 2023-10-30T12:19:26.925 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: id: e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:19:26.925 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: health: HEALTH_WARN 2023-10-30T12:19:26.925 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 5 pool(s) do not have an application enabled 2023-10-30T12:19:26.925 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:26.925 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: services: 2023-10-30T12:19:26.926 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-10-30T12:19:26.926 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: mgr: y(active, since 2m), standbys: x 2023-10-30T12:19:26.926 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: osd: 8 osds: 8 up (since 93s), 8 in (since 2m) 2023-10-30T12:19:26.926 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-10-30T12:19:26.926 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: data: 2023-10-30T12:19:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: pools: 24 pools, 713 pgs 2023-10-30T12:19:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: objects: 2 objects, 449 KiB 2023-10-30T12:19:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-10-30T12:19:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: pgs: 84.712% pgs unknown 2023-10-30T12:19:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 14.025% pgs not active 2023-10-30T12:19:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 604 unknown 2023-10-30T12:19:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 100 creating+peering 2023-10-30T12:19:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 9 active+clean 2023-10-30T12:19:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:26.929 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: cluster: 2023-10-30T12:19:26.929 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: id: e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:19:26.929 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: health: HEALTH_WARN 2023-10-30T12:19:26.929 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 8 pool(s) do not have an application enabled 2023-10-30T12:19:26.929 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:26.930 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: services: 2023-10-30T12:19:26.930 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-10-30T12:19:26.930 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: mgr: y(active, since 2m), standbys: x 2023-10-30T12:19:26.930 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: osd: 8 osds: 8 up (since 95s), 8 in (since 2m) 2023-10-30T12:19:26.930 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-10-30T12:19:26.930 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-10-30T12:19:26.931 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:26.931 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: data: 2023-10-30T12:19:26.931 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: pools: 30 pools, 881 pgs 2023-10-30T12:19:26.931 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: objects: 4 objects, 449 KiB 2023-10-30T12:19:26.931 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-10-30T12:19:26.932 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: pgs: 17.934% pgs unknown 2023-10-30T12:19:26.932 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 73.439% pgs not active 2023-10-30T12:19:28.529 INFO:tasks.workunit.client.0.smithi148.stdout: 360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.530 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:21.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.530 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:22.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:28.530 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:22.834+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.530 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:22.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.531 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:23.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:28.531 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:23.834+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.531 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:23.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.531 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:24.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:28.531 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:24.834+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.532 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:24.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.532 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:25.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:28.532 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:25.834+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.532 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:25.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.532 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:25.999+0000 7fd366474640 10 client.4379.objecter tick 2023-10-30T12:19:28.533 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:26.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:28.533 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:26.834+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.533 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:26.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.533 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:27.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:28.533 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:27.834+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.534 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:27.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.534 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 2 ==== osd_op_reply(2 exist [writefull 0~5] v28'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:28.534 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 1 ==== osd_map(27..28 src has 1..28) v4 ==== 6007+0+0 (crc 0 0 0) 0x7fd35c005b00 con 0x559edb509a60 2023-10-30T12:19:28.534 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_map(27..28 src has 1..28) v4 2023-10-30T12:19:28.534 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(2 exist [writefull 0~5] v28'1 uv1 ondisk = 0) v8 2023-10-30T12:19:28.535 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:28.535 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map got epochs [27,28] > 26 2023-10-30T12:19:28.535 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 27 2023-10-30T12:19:28.535 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 10 client.4379.objecter checking op 2 2023-10-30T12:19:28.535 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 27 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:28.535 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:28.536 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 28 2023-10-30T12:19:28.536 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 10 client.4379.objecter checking op 2 2023-10-30T12:19:28.536 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 28 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:28.536 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:28.536 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 20 client.4379.objecter dump_active .. 0 homeless 2023-10-30T12:19:28.537 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd361a0b640 20 client.4379.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2023-10-30T12:19:28.537 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 2 ondisk uv 1 in 20.1 attempt 0 2023-10-30T12:19:28.537 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:28.537 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 2 2023-10-30T12:19:28.537 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd364a11640 15 client.4379.objecter _finish_op 2 2023-10-30T12:19:28.538 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:28.538 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 2 2023-10-30T12:19:28.538 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:28.538 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-10-30T12:19:28.538 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 10 librados: Objecter returned from writefull r=0 2023-10-30T12:19:28.538 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5a4f50 2023-10-30T12:19:28.539 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 28 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:28.539 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:28.539 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:28.539 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:28.539 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [read 0~256] tid 3 osd.0 2023-10-30T12:19:28.540 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:28.540 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 3 2023-10-30T12:19:28.540 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 15 client.4379.objecter _send_op 3 to 20.1 on osd.0 2023-10-30T12:19:28.540 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:3 20.1 20:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) v8 -- 0x559edb4f89a0 con 0x559edb509a60 2023-10-30T12:19:28.540 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:28.541 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:28.541 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb4f8d50 2023-10-30T12:19:28.541 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 28 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:28.541 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:28.541 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:32.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:32.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 10 client.4379.objecter _op_submit oid noexist '@20' '@20' [read 0~256] tid 4 osd.0 2023-10-30T12:19:32.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:32.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 4 2023-10-30T12:19:32.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 15 client.4379.objecter _send_op 4 to 20.1 on osd.0 2023-10-30T12:19:32.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:4 20.1 20:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) v8 -- 0x559edb4f91f0 con 0x559edb509a60 2023-10-30T12:19:32.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:32.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.528+0000 7fd366e44780 5 client.4379.objecter 2 in flight 2023-10-30T12:19:32.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:32.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.834+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:32.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:32.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:28.834+0000 7fd360a09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:32.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:29.834+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:32.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:29.834+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:32.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:29.834+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:32.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:30.835+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:32.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:30.835+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:32.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:30.835+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:32.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:31.000+0000 7fd366474640 10 client.4379.objecter tick 2023-10-30T12:19:32.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:31.836+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:32.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:31.836+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:32.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:31.836+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:32.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 3 ==== osd_map(29..33 src has 1..35) v4 ==== 4550+0+0 (crc 0 0 0) 0x7fd35c00ecc0 con 0x559edb509a60 2023-10-30T12:19:32.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:32.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_map(29..33 src has 1..35) v4 2023-10-30T12:19:32.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T12:19:32.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:32.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map got epochs [29,33] > 28 2023-10-30T12:19:32.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 29 2023-10-30T12:19:32.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter checking op 3 2023-10-30T12:19:32.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 29 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:32.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter checking op 4 2023-10-30T12:19:32.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 29 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:32.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 30 2023-10-30T12:19:32.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter checking op 3 2023-10-30T12:19:32.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 30 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:32.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter checking op 4 2023-10-30T12:19:32.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 30 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:32.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 31 2023-10-30T12:19:32.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter checking op 3 2023-10-30T12:19:32.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 31 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:32.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter checking op 4 2023-10-30T12:19:32.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 31 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:32.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 32 2023-10-30T12:19:32.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter checking op 3 2023-10-30T12:19:32.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 32 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:32.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter checking op 4 2023-10-30T12:19:32.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 32 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:32.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 33 2023-10-30T12:19:32.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter checking op 3 2023-10-30T12:19:32.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 33 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:32.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 10 client.4379.objecter checking op 4 2023-10-30T12:19:32.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 33 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:32.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter dump_active .. 0 homeless 2023-10-30T12:19:32.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter 3 20.d3a59201 osd.0 exist [read 0~256] 2023-10-30T12:19:32.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd361a0b640 20 client.4379.objecter 4 20.a7bbaa1 osd.0 noexist [read 0~256] 2023-10-30T12:19:32.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 3 ondisk uv 1 in 20.1 attempt 0 2023-10-30T12:19:32.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 5 2023-10-30T12:19:32.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 3 2023-10-30T12:19:32.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 15 client.4379.objecter _finish_op 3 2023-10-30T12:19:32.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:32.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 3 2023-10-30T12:19:32.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 5 client.4379.objecter 1 in flight 2023-10-30T12:19:32.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:32.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:19:32.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:32.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 4 ondisk uv 0 in 20.1 attempt 0 2023-10-30T12:19:32.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:19:32.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:32.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 4 2023-10-30T12:19:32.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 15 client.4379.objecter _finish_op 4 2023-10-30T12:19:32.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:32.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 4 2023-10-30T12:19:32.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:32.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (3918 ms) 2023-10-30T12:19:32.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-10-30T12:19:32.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb4f8d50 2023-10-30T12:19:32.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 33 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:32.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:32.468 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:32.831 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [read 0~256] tid 5 osd.0 2023-10-30T12:19:32.832 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:32.832 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 5 2023-10-30T12:19:32.832 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 15 client.4379.objecter _send_op 5 to 20.1 on osd.0 2023-10-30T12:19:32.833 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:5 20.1 20:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x559edb4f95b0 con 0x559edb509a60 2023-10-30T12:19:32.833 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:32.833 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:32.833 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5a4f50 2023-10-30T12:19:32.833 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 33 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.833 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:32.834 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:32.834 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:32.834 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 10 client.4379.objecter _op_submit oid noexist '@20' '@20' [read 0~256] tid 6 osd.0 2023-10-30T12:19:32.834 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:32.834 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 6 2023-10-30T12:19:32.835 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 15 client.4379.objecter _send_op 6 to 20.1 on osd.0 2023-10-30T12:19:32.835 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:6 20.1 20:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x559edb5c0330 con 0x559edb509a60 2023-10-30T12:19:32.835 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:32.835 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.447+0000 7fd366e44780 5 client.4379.objecter 2 in flight 2023-10-30T12:19:32.835 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 6 ==== osd_map(34..35 src has 1..35) v4 ==== 1311+0+0 (crc 0 0 0) 0x7fd35c0129f0 con 0x559edb509a60 2023-10-30T12:19:32.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 7 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:32.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_map(34..35 src has 1..35) v4 2023-10-30T12:19:32.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T12:19:32.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:32.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map got epochs [34,35] > 33 2023-10-30T12:19:32.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 34 2023-10-30T12:19:32.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 10 client.4379.objecter checking op 5 2023-10-30T12:19:32.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 34 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:32.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 10 client.4379.objecter checking op 6 2023-10-30T12:19:32.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 34 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:32.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 35 2023-10-30T12:19:32.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 10 client.4379.objecter checking op 5 2023-10-30T12:19:32.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 35 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:32.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 10 client.4379.objecter checking op 6 2023-10-30T12:19:32.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 35 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:32.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter dump_active .. 0 homeless 2023-10-30T12:19:32.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter 5 20.d3a59201 osd.0 exist [read 0~256] 2023-10-30T12:19:32.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd361a0b640 20 client.4379.objecter 6 20.a7bbaa1 osd.0 noexist [read 0~256] 2023-10-30T12:19:32.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 5 ondisk uv 1 in 20.1 attempt 0 2023-10-30T12:19:32.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 5 2023-10-30T12:19:32.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 5 2023-10-30T12:19:32.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 15 client.4379.objecter _finish_op 5 2023-10-30T12:19:32.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:32.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 5 2023-10-30T12:19:32.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 5 client.4379.objecter 1 in flight 2023-10-30T12:19:32.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 8 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:32.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:19:32.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:32.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 6 ondisk uv 0 in 20.1 attempt 0 2023-10-30T12:19:32.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:19:32.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:32.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 6 2023-10-30T12:19:32.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 15 client.4379.objecter _finish_op 6 2023-10-30T12:19:32.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:32.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 6 2023-10-30T12:19:32.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:32.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (384 ms) 2023-10-30T12:19:32.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-10-30T12:19:32.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5a4f50 2023-10-30T12:19:32.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 35 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:32.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:32.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:32.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [read 0~256] tid 7 osd.0 2023-10-30T12:19:32.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:32.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 7 2023-10-30T12:19:32.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 15 client.4379.objecter _send_op 7 to 20.1 on osd.0 2023-10-30T12:19:32.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:7 20.1 20:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v8 -- 0x559edb4f8d30 con 0x559edb509a60 2023-10-30T12:19:32.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:32.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:32.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb600d30 2023-10-30T12:19:32.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 35 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:32.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:32.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:32.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:32.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 10 client.4379.objecter _op_submit oid noexist '@20' '@20' [read 0~256] tid 8 osd.0 2023-10-30T12:19:32.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:32.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 8 2023-10-30T12:19:32.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 15 client.4379.objecter _send_op 8 to 20.1 on osd.0 2023-10-30T12:19:32.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:8 20.1 20:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v8 -- 0x559edb6011d0 con 0x559edb509a60 2023-10-30T12:19:34.135 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:34.135 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.830+0000 7fd366e44780 5 client.4379.objecter 2 in flight 2023-10-30T12:19:34.135 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.835+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:34.136 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.835+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:34.136 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:32.835+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:34.136 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:33.835+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:34.136 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:33.835+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:34.136 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:33.835+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:34.136 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 9 ==== osd_map(36..36 src has 1..36) v4 ==== 636+0+0 (crc 0 0 0) 0x7fd35c012bc0 con 0x559edb509a60 2023-10-30T12:19:34.137 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 10 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:34.137 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_map(36..36 src has 1..36) v4 2023-10-30T12:19:34.137 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T12:19:34.137 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:34.137 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map got epochs [36,36] > 35 2023-10-30T12:19:34.138 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 36 2023-10-30T12:19:34.138 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 10 client.4379.objecter checking op 7 2023-10-30T12:19:34.138 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 36 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:34.138 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:34.138 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 10 client.4379.objecter checking op 8 2023-10-30T12:19:34.139 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 36 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:34.139 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:34.139 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 20 client.4379.objecter dump_active .. 0 homeless 2023-10-30T12:19:34.139 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1303 ms) 2023-10-30T12:19:34.139 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 20 client.4379.objecter 7 20.d3a59201 osd.0 exist [read 0~256] 2023-10-30T12:19:34.139 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-10-30T12:19:34.140 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd361a0b640 20 client.4379.objecter 8 20.a7bbaa1 osd.0 noexist [read 0~256] 2023-10-30T12:19:34.140 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 7 ondisk uv 1 in 20.1 attempt 0 2023-10-30T12:19:34.140 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 5 2023-10-30T12:19:34.140 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 7 2023-10-30T12:19:34.140 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 15 client.4379.objecter _finish_op 7 2023-10-30T12:19:34.141 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:34.836 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 7 2023-10-30T12:19:34.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 5 client.4379.objecter 1 in flight 2023-10-30T12:19:34.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 11 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:34.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:19:34.837 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:34.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 8 ondisk uv 0 in 20.1 attempt 0 2023-10-30T12:19:34.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:19:34.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:34.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 8 2023-10-30T12:19:34.838 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 15 client.4379.objecter _finish_op 8 2023-10-30T12:19:34.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:34.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 8 2023-10-30T12:19:34.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:34.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T12:19:34.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 20 librados: queue_aio_write 0x559edb390db0 completion 0x559edb5a4f50 write_seq 1 2023-10-30T12:19:34.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c16b0 2023-10-30T12:19:34.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 36 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:34.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:34.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:34.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:34.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [write 0~5 in=5b] tid 9 osd.0 2023-10-30T12:19:34.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:34.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 9 2023-10-30T12:19:34.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 15 client.4379.objecter _send_op 9 to 20.1 on osd.0 2023-10-30T12:19:34.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:9 20.1 20:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) v8 -- 0x559edb5a5060 con 0x559edb509a60 2023-10-30T12:19:34.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:34.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:34.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.133+0000 7fd366e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T12:19:35.401 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.835+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:35.401 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.835+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:35.401 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:34.835+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:35.402 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.120+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 12 ==== osd_map(37..37 src has 1..37) v4 ==== 304+0+0 (crc 0 0 0) 0x7fd35c012470 con 0x559edb509a60 2023-10-30T12:19:35.402 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.120+0000 7fd361a0b640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_map(37..37 src has 1..37) v4 2023-10-30T12:19:35.402 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.121+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map got epochs [37,37] > 36 2023-10-30T12:19:35.402 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.121+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 37 2023-10-30T12:19:35.402 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.121+0000 7fd361a0b640 10 client.4379.objecter checking op 9 2023-10-30T12:19:35.403 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.121+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 37 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:35.403 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.121+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:35.403 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.121+0000 7fd361a0b640 20 client.4379.objecter dump_active .. 0 homeless 2023-10-30T12:19:35.403 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.121+0000 7fd361a0b640 20 client.4379.objecter 9 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2023-10-30T12:19:35.403 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 13 ==== osd_op_reply(9 exist [write 0~5] v37'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:35.404 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(9 exist [write 0~5] v37'2 uv2 ondisk = 0) v8 2023-10-30T12:19:35.404 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:35.404 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 9 ondisk uv 2 in 20.1 attempt 0 2023-10-30T12:19:35.404 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:35.404 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 9 2023-10-30T12:19:35.405 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 15 client.4379.objecter _finish_op 9 2023-10-30T12:19:35.405 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.405 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 9 2023-10-30T12:19:35.405 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:35.405 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd364a11640 20 librados: complete_aio_write 0x559edb5a4f50 2023-10-30T12:19:35.405 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (1266 ms) 2023-10-30T12:19:35.406 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-10-30T12:19:35.406 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T12:19:35.406 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 20 librados: queue_aio_write 0x559edb390db0 completion 0x559edb5c1b50 write_seq 2 2023-10-30T12:19:35.406 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c16b0 2023-10-30T12:19:35.406 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:35.407 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:35.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:35.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [write 0~5 in=5b] tid 10 osd.0 2023-10-30T12:19:35.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 10 2023-10-30T12:19:35.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 15 client.4379.objecter _send_op 10 to 20.1 on osd.0 2023-10-30T12:19:35.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:10 20.1 20:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5a5060 con 0x559edb509a60 2023-10-30T12:19:35.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:35.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.399+0000 7fd366e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T12:19:35.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 14 ==== osd_op_reply(10 exist [write 0~5] v37'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:35.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(10 exist [write 0~5] v37'3 uv3 ondisk = 0) v8 2023-10-30T12:19:35.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:35.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 10 ondisk uv 3 in 20.1 attempt 0 2023-10-30T12:19:35.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:35.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 10 2023-10-30T12:19:35.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 15 client.4379.objecter _finish_op 10 2023-10-30T12:19:35.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.631 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 10 2023-10-30T12:19:35.631 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:35.631 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd364a11640 20 librados: complete_aio_write 0x559edb5c1b50 2023-10-30T12:19:35.631 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (226 ms) 2023-10-30T12:19:35.631 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-10-30T12:19:35.631 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T12:19:35.632 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 20 librados: queue_aio_write 0x559edb390db0 completion 0x559edb5a5240 write_seq 3 2023-10-30T12:19:35.632 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c16b0 2023-10-30T12:19:35.632 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:35.632 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:35.632 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:35.633 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.795 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [write 0~5 in=5b] tid 11 osd.0 2023-10-30T12:19:35.795 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.795 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 11 2023-10-30T12:19:35.796 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 15 client.4379.objecter _send_op 11 to 20.1 on osd.0 2023-10-30T12:19:35.796 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:11 20.1 20:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5e43a0 con 0x559edb509a60 2023-10-30T12:19:35.796 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.796 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.625+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:35.796 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.626+0000 7fd366e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T12:19:35.797 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 15 ==== osd_op_reply(11 exist [write 0~5] v37'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:35.797 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(11 exist [write 0~5] v37'4 uv4 ondisk = 0) v8 2023-10-30T12:19:35.797 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:35.797 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 11 ondisk uv 4 in 20.1 attempt 0 2023-10-30T12:19:35.797 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:35.798 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 11 2023-10-30T12:19:35.798 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 15 client.4379.objecter _finish_op 11 2023-10-30T12:19:35.798 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.798 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 11 2023-10-30T12:19:35.798 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:35.799 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd364a11640 20 librados: complete_aio_write 0x559edb5a5240 2023-10-30T12:19:35.799 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (168 ms) 2023-10-30T12:19:35.799 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-10-30T12:19:35.799 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c06a0 2023-10-30T12:19:35.799 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:35.800 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:35.800 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:35.800 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.800 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [read 0~0] tid 12 osd.0 2023-10-30T12:19:35.800 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.801 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 12 2023-10-30T12:19:35.808 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 15 client.4379.objecter _send_op 12 to 20.1 on osd.0 2023-10-30T12:19:35.808 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:12 20.1 20:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5c0b40 con 0x559edb509a60 2023-10-30T12:19:35.808 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.809 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:35.809 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c0fc0 2023-10-30T12:19:35.809 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:35.809 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:35.809 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:35.810 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.810 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 10 client.4379.objecter _op_submit oid noexist '@20' '@20' [read 0~0] tid 13 osd.0 2023-10-30T12:19:35.810 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.810 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 13 2023-10-30T12:19:35.810 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 15 client.4379.objecter _send_op 13 to 20.1 on osd.0 2023-10-30T12:19:35.811 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:13 20.1 20:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5c1460 con 0x559edb509a60 2023-10-30T12:19:35.811 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:35.811 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.793+0000 7fd366e44780 5 client.4379.objecter 2 in flight 2023-10-30T12:19:35.811 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 16 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:35.811 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T12:19:35.812 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:35.812 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 12 ondisk uv 4 in 20.1 attempt 0 2023-10-30T12:19:35.812 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 5 2023-10-30T12:19:35.812 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 12 2023-10-30T12:19:35.812 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 15 client.4379.objecter _finish_op 12 2023-10-30T12:19:35.813 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:35.813 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 12 2023-10-30T12:19:35.813 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 5 client.4379.objecter 1 in flight 2023-10-30T12:19:35.813 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 17 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:35.813 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:19:35.814 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:35.814 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 13 ondisk uv 0 in 20.1 attempt 0 2023-10-30T12:19:35.814 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:19:35.814 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:35.814 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 13 2023-10-30T12:19:35.815 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 15 client.4379.objecter _finish_op 13 2023-10-30T12:19:35.815 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.815 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 13 2023-10-30T12:19:35.815 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.806+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:35.815 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (14 ms) 2023-10-30T12:19:35.816 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-10-30T12:19:35.816 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c07b0 2023-10-30T12:19:35.816 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:35.816 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:35.816 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:35.816 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.817 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [read 0~0] tid 14 osd.0 2023-10-30T12:19:35.817 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.817 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 14 2023-10-30T12:19:35.817 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 15 client.4379.objecter _send_op 14 to 20.1 on osd.0 2023-10-30T12:19:35.817 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:14 20.1 20:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5c0fc0 con 0x559edb509a60 2023-10-30T12:19:35.818 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.818 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:35.818 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c14c0 2023-10-30T12:19:35.818 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:35.818 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:35.819 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:35.819 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.828 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 10 client.4379.objecter _op_submit oid noexist '@20' '@20' [read 0~0] tid 15 osd.0 2023-10-30T12:19:35.828 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.828 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 15 2023-10-30T12:19:35.828 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 15 client.4379.objecter _send_op 15 to 20.1 on osd.0 2023-10-30T12:19:35.829 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:15 20.1 20:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5e48e0 con 0x559edb509a60 2023-10-30T12:19:35.829 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:35.829 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.807+0000 7fd366e44780 5 client.4379.objecter 2 in flight 2023-10-30T12:19:35.829 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 18 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:35.829 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T12:19:35.830 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:35.830 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 14 ondisk uv 4 in 20.1 attempt 0 2023-10-30T12:19:35.830 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 5 2023-10-30T12:19:35.830 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 14 2023-10-30T12:19:35.830 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 15 client.4379.objecter _finish_op 14 2023-10-30T12:19:35.830 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:35.831 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 14 2023-10-30T12:19:35.831 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 5 client.4379.objecter 1 in flight 2023-10-30T12:19:35.831 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 19 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:35.831 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:19:35.831 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:35.832 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 15 ondisk uv 0 in 20.1 attempt 0 2023-10-30T12:19:35.832 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:19:35.832 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:35.832 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 15 2023-10-30T12:19:35.832 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 15 client.4379.objecter _finish_op 15 2023-10-30T12:19:35.833 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.833 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 15 2023-10-30T12:19:35.833 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:35.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (19 ms) 2023-10-30T12:19:35.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-10-30T12:19:35.839 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c14c0 2023-10-30T12:19:35.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:35.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:35.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:35.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.840 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [read 0~0] tid 16 osd.0 2023-10-30T12:19:35.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 16 2023-10-30T12:19:35.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 15 client.4379.objecter _send_op 16 to 20.1 on osd.0 2023-10-30T12:19:35.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:16 20.1 20:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5c0fc0 con 0x559edb509a60 2023-10-30T12:19:35.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:35.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5e3d30 2023-10-30T12:19:35.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:19:35.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 20 client.4379.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2023-10-30T12:19:35.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:35.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 10 client.4379.objecter _op_submit oid noexist '@20' '@20' [read 0~0] tid 17 osd.0 2023-10-30T12:19:35.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 17 2023-10-30T12:19:35.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 15 client.4379.objecter _send_op 17 to 20.1 on osd.0 2023-10-30T12:19:35.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:17 20.1 20:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5e41d0 con 0x559edb509a60 2023-10-30T12:19:35.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:35.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.826+0000 7fd366e44780 5 client.4379.objecter 2 in flight 2023-10-30T12:19:35.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.835+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:35.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.835+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:35.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.835+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:35.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 20 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:35.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T12:19:35.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:35.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 16 ondisk uv 4 in 20.1 attempt 0 2023-10-30T12:19:35.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 5 2023-10-30T12:19:35.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 16 2023-10-30T12:19:35.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 15 client.4379.objecter _finish_op 16 2023-10-30T12:19:35.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 5 2023-10-30T12:19:35.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 16 2023-10-30T12:19:35.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 5 client.4379.objecter 1 in flight 2023-10-30T12:19:35.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 21 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:35.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:19:35.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:35.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 17 ondisk uv 0 in 20.1 attempt 0 2023-10-30T12:19:35.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:19:35.848 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:35.848 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 17 2023-10-30T12:19:35.848 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 15 client.4379.objecter _finish_op 17 2023-10-30T12:19:35.848 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.848 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 17 2023-10-30T12:19:35.849 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:35.849 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (11 ms) 2023-10-30T12:19:35.849 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-10-30T12:19:35.849 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 20 librados: queue_aio_write 0x559edb390db0 completion 0x559edb5e48c0 write_seq 4 2023-10-30T12:19:35.849 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c14c0 2023-10-30T12:19:35.849 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:35.850 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:35.850 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:35.850 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.883 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [writefull 0~5 in=5b] tid 18 osd.0 2023-10-30T12:19:35.883 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.883 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 18 2023-10-30T12:19:35.883 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 15 client.4379.objecter _send_op 18 to 20.1 on osd.0 2023-10-30T12:19:35.884 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.837+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:18 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5c0fc0 con 0x559edb509a60 2023-10-30T12:19:35.884 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.838+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.884 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.838+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:35.884 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 22 ==== osd_op_reply(18 exist [writefull 0~5] v37'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:35.885 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(18 exist [writefull 0~5] v37'5 uv5 ondisk = 0) v8 2023-10-30T12:19:35.885 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:35.885 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 18 ondisk uv 5 in 20.1 attempt 0 2023-10-30T12:19:35.885 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:35.886 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 18 2023-10-30T12:19:35.886 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 15 client.4379.objecter _finish_op 18 2023-10-30T12:19:35.886 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:35.886 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 18 2023-10-30T12:19:35.887 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:35.887 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd364a11640 20 librados: complete_aio_write 0x559edb5e48c0 2023-10-30T12:19:35.887 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (44 ms) 2023-10-30T12:19:35.887 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-10-30T12:19:35.887 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 20 librados: queue_aio_write 0x559edb390db0 completion 0x559edb5e45c0 write_seq 5 2023-10-30T12:19:35.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c14c0 2023-10-30T12:19:35.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:35.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:35.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:35.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [writefull 0~5 in=5b] tid 19 osd.0 2023-10-30T12:19:35.889 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:35.889 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 19 2023-10-30T12:19:36.540 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 15 client.4379.objecter _send_op 19 to 20.1 on osd.0 2023-10-30T12:19:36.540 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:19 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5c0fc0 con 0x559edb509a60 2023-10-30T12:19:36.540 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:36.540 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:35.881+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:36.541 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.000+0000 7fd366474640 10 client.4379.objecter tick 2023-10-30T12:19:36.541 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 23 ==== osd_op_reply(19 exist [writefull 0~5] v37'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:36.541 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(19 exist [writefull 0~5] v37'6 uv6 ondisk = 0) v8 2023-10-30T12:19:36.541 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:36.541 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 19 ondisk uv 6 in 20.1 attempt 0 2023-10-30T12:19:36.542 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:36.542 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 19 2023-10-30T12:19:36.542 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 15 client.4379.objecter _finish_op 19 2023-10-30T12:19:36.542 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:36.542 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 19 2023-10-30T12:19:36.543 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:36.543 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd364a11640 20 librados: complete_aio_write 0x559edb5e45c0 2023-10-30T12:19:36.543 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (657 ms) 2023-10-30T12:19:36.544 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-10-30T12:19:36.544 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 20 librados: queue_aio_write 0x559edb390db0 completion 0x559edb5e48c0 write_seq 6 2023-10-30T12:19:36.544 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 10 client.4379.objecter _op_submit op 0x559edb5c14c0 2023-10-30T12:19:36.544 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 20 client.4379.objecter _calc_target epoch 37 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:36.545 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:36.545 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 10 client.4379.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2023-10-30T12:19:36.545 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 20 client.4379.objecter _get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:36.545 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 10 client.4379.objecter _op_submit oid exist '@20' '@20' [writefull 0~5 in=5b] tid 20 osd.0 2023-10-30T12:19:36.546 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 20 client.4379.objecter get_session s=0x559edb43bb70 osd=0 3 2023-10-30T12:19:36.546 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 15 client.4379.objecter _session_op_assign 0 20 2023-10-30T12:19:36.546 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 15 client.4379.objecter _send_op 20 to 20.1 on osd.0 2023-10-30T12:19:37.765 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 --> v2:172.21.15.148:6805/2395694124 -- osd_op(unknown.0.0:20 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) v8 -- 0x559edb5c0fc0 con 0x559edb509a60 2023-10-30T12:19:37.765 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:37.765 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.538+0000 7fd366e44780 5 client.4379.objecter 1 in flight 2023-10-30T12:19:37.765 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.625+0000 7fd361a0b640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 24 ==== osd_map(38..38 src has 1..38) v4 ==== 320+0+0 (crc 0 0 0) 0x7fd35c012640 con 0x559edb509a60 2023-10-30T12:19:37.766 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.625+0000 7fd361a0b640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_map(38..38 src has 1..38) v4 2023-10-30T12:19:37.766 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.625+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map got epochs [38,38] > 37 2023-10-30T12:19:37.766 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.625+0000 7fd361a0b640 3 client.4379.objecter handle_osd_map decoding incremental epoch 38 2023-10-30T12:19:37.766 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.625+0000 7fd361a0b640 10 client.4379.objecter checking op 20 2023-10-30T12:19:37.766 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.625+0000 7fd361a0b640 20 client.4379.objecter _calc_target epoch 38 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:19:37.767 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.625+0000 7fd361a0b640 20 client.4379.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2023-10-30T12:19:37.767 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.625+0000 7fd361a0b640 20 client.4379.objecter dump_active .. 0 homeless 2023-10-30T12:19:37.767 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.625+0000 7fd361a0b640 20 client.4379.objecter 20 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2023-10-30T12:19:37.767 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.835+0000 7fd360a09640 10 monclient: tick 2023-10-30T12:19:37.767 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.835+0000 7fd360a09640 10 monclient: _check_auth_tickets 2023-10-30T12:19:37.768 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:36.835+0000 7fd360a09640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:37.768 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 <== osd.0 v2:172.21.15.148:6805/2395694124 25 ==== osd_op_reply(20 exist [writefull 0~5] v38'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fd35c018070 con 0x559edb509a60 2023-10-30T12:19:37.768 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 10 client.4379.objecter ms_dispatch 0x559edb3c7d50 osd_op_reply(20 exist [writefull 0~5] v38'7 uv7 ondisk = 0) v8 2023-10-30T12:19:37.768 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 10 client.4379.objecter in handle_osd_op_reply 2023-10-30T12:19:37.768 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 7 client.4379.objecter handle_osd_op_reply 20 ondisk uv 7 in 20.1 attempt 0 2023-10-30T12:19:37.769 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 10 client.4379.objecter op 0 rval 0 len 0 2023-10-30T12:19:37.769 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 15 client.4379.objecter handle_osd_op_reply completed tid 20 2023-10-30T12:19:37.769 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 15 client.4379.objecter _finish_op 20 2023-10-30T12:19:37.769 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 4 2023-10-30T12:19:37.769 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 15 client.4379.objecter _session_op_remove 0 20 2023-10-30T12:19:37.770 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 5 client.4379.objecter 0 in flight 2023-10-30T12:19:37.770 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 20 librados: complete_aio_write 0x559edb5e48c0 2023-10-30T12:19:37.770 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1225 ms) 2023-10-30T12:19:37.770 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 10 librados: watch_flush enter 2023-10-30T12:19:37.771 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 10 librados: watch_flush exit 2023-10-30T12:19:37.771 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 10 client.4379.objecter close_session for osd.0 2023-10-30T12:19:37.771 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 >> v2:172.21.15.148:6805/2395694124 conn(0x559edb509a60 msgr2=0x559edb5a5900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:37.771 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 --2- 172.21.15.148:0/2607214554 >> v2:172.21.15.148:6805/2395694124 conn(0x559edb509a60 0x559edb5a5900 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:37.772 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 20 client.4379.objecter put_session s=0x559edb43bb70 osd=0 1 2023-10-30T12:19:37.772 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 20 client.4379.objecter shutdown clearing up homeless session... 2023-10-30T12:19:37.772 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 10 client.4379.objecter successfully canceled tick 2023-10-30T12:19:37.772 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd35006a9b0 msgr2=0x7fd35006ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:37.773 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 --2- 172.21.15.148:0/2607214554 >> v2:172.21.15.148:6800/3205090179 conn(0x7fd35006a9b0 0x7fd35006ce70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x559edb440f10 tx=0x7fd358052000 comp rx=0 tx=0).stop 2023-10-30T12:19:37.773 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 10 monclient: shutdown 2023-10-30T12:19:37.773 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:37.773 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x559edb4415a0 msgr2=0x559edb505440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:37.773 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 --2- 172.21.15.148:0/2607214554 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x559edb4415a0 0x559edb505440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd35400aec0 tx=0x7fd3540086e0 comp rx=0 tx=0).stop 2023-10-30T12:19:37.774 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd364a11640 1 -- 172.21.15.148:0/2607214554 reap_dead start 2023-10-30T12:19:37.774 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 shutdown_connections 2023-10-30T12:19:37.774 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 >> 172.21.15.148:0/2607214554 conn(0x559edb43b750 msgr2=0x559edb47f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:37.774 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 shutdown_connections 2023-10-30T12:19:37.774 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 -- 172.21.15.148:0/2607214554 wait complete. 2023-10-30T12:19:37.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.763+0000 7fd366e44780 1 librados: shutdown 2023-10-30T12:19:37.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [----------] 12 tests from AsioRados (9235 ms total) 2023-10-30T12:19:37.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: 2023-10-30T12:19:37.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [----------] Global test environment tear-down 2023-10-30T12:19:37.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (22134 ms total) 2023-10-30T12:19:37.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_asio: [ PASSED ] 12 tests. 2023-10-30T12:19:41.667 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:02:56.740 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:41.673 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:02:56.747 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:41.722 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:56.796 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:41.762 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:02:56.809 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:41.766 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:02:56.813 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:41.824 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:56.897 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:41.854 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:02:56.928 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:41.910 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:02:56.957 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:42.001 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:57.073 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:47.015 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 353 creating+activating 2023-10-30T12:19:47.015 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 294 creating+peering 2023-10-30T12:19:47.015 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 158 unknown 2023-10-30T12:19:47.015 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 76 active+clean 2023-10-30T12:19:47.016 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:47.016 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: io: 2023-10-30T12:19:47.016 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: client: 384 B/s wr, 0 op/s rd, 0 op/s wr 2023-10-30T12:19:47.016 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:47.017 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:47.017 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5508 ms) 2023-10-30T12:19:47.017 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [ RUN ] LibRadosService.Status 2023-10-30T12:19:47.017 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [ OK ] LibRadosService.Status (20059 ms) 2023-10-30T12:19:47.017 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [----------] 4 tests from LibRadosService (31236 ms total) 2023-10-30T12:19:47.018 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: 2023-10-30T12:19:47.018 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [----------] Global test environment tear-down 2023-10-30T12:19:47.018 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31236 ms total) 2023-10-30T12:19:47.018 INFO:tasks.workunit.client.0.smithi148.stdout: api_service: [ PASSED ] 4 tests. 2023-10-30T12:19:47.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-10-30T12:19:47.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [----------] Global test environment set-up. 2023-10-30T12:19:47.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-10-30T12:19:47.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-10-30T12:19:47.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5356 ms) 2023-10-30T12:19:47.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-10-30T12:19:47.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (361 ms) 2023-10-30T12:19:47.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-10-30T12:19:47.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20398 ms) 2023-10-30T12:19:47.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-10-30T12:19:47.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: attempt 0 of 20 2023-10-30T12:19:47.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: failed to deregister: 2023-10-30T12:19:47.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: { 2023-10-30T12:19:47.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "epoch": 10, 2023-10-30T12:19:47.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "modified": "2023-10-30T12:19:42.993290+0000", 2023-10-30T12:19:47.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "services": { 2023-10-30T12:19:47.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "laundry": { 2023-10-30T12:19:47.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "daemons": { 2023-10-30T12:19:47.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "summary": "", 2023-10-30T12:19:47.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "close-test-pid92539": { 2023-10-30T12:19:47.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "start_epoch": 10, 2023-10-30T12:19:47.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "start_stamp": "2023-10-30T12:19:42.331973+0000", 2023-10-30T12:19:47.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "gid": 4544, 2023-10-30T12:19:47.486 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "addr": "172.21.15.148:0/4232027625", 2023-10-30T12:19:47.486 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "metadata": { 2023-10-30T12:19:47.486 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "arch": "x86_64", 2023-10-30T12:19:47.486 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "ceph_release": "reef", 2023-10-30T12:19:47.486 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-6839-gcc36dd85 (cc36dd85d142c6fdb77515cad8af108cec98a86d) reef (dev)", 2023-10-30T12:19:47.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "ceph_version_short": "18.0.0-6839-gcc36dd85", 2023-10-30T12:19:47.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-10-30T12:19:47.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "distro": "centos", 2023-10-30T12:19:47.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-10-30T12:19:47.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "distro_version": "9", 2023-10-30T12:19:47.488 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "foo": "bar", 2023-10-30T12:19:47.488 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "hostname": "smithi148", 2023-10-30T12:19:47.488 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-10-30T12:19:47.488 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "kernel_version": "5.14.0-378.el9.x86_64", 2023-10-30T12:19:47.488 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "mem_swap_kb": "0", 2023-10-30T12:19:47.489 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "mem_total_kb": "32488716", 2023-10-30T12:19:47.489 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "os": "Linux", 2023-10-30T12:19:47.489 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "this": "that" 2023-10-30T12:19:47.489 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: }, 2023-10-30T12:19:47.489 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "task_status": {} 2023-10-30T12:19:47.490 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: }, 2023-10-30T12:19:47.490 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "pid92496": { 2023-10-30T12:19:47.490 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "start_epoch": 9, 2023-10-30T12:19:47.490 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "start_stamp": "2023-10-30T12:19:27.023381+0000", 2023-10-30T12:19:47.490 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "gid": 4672, 2023-10-30T12:19:47.491 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "addr": "172.21.15.148:0/3202305948", 2023-10-30T12:19:47.491 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "metadata": { 2023-10-30T12:19:47.491 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "arch": "x86_64", 2023-10-30T12:19:47.491 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "ceph_release": "reef", 2023-10-30T12:19:47.491 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-6839-gcc36dd85 (cc36dd85d142c6fdb77515cad8af108cec98a86d) reef (dev)", 2023-10-30T12:19:47.492 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "ceph_version_short": "18.0.0-6839-gcc36dd85", 2023-10-30T12:19:47.492 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-10-30T12:19:52.803 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "distro": "centos", 2023-10-30T12:19:52.803 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-10-30T12:19:52.803 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "distro_version": "9", 2023-10-30T12:19:52.803 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "foo": "bar", 2023-10-30T12:19:52.804 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "hostname": "smithi148", 2023-10-30T12:19:52.804 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-10-30T12:19:52.804 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "kernel_version": "5.14.0-378.el9.x86_64", 2023-10-30T12:19:52.804 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "mem_swap_kb": "0", 2023-10-30T12:19:52.804 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "mem_total_kb": "32488716", 2023-10-30T12:19:52.804 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "os": "Linux", 2023-10-30T12:19:52.805 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "this": "that" 2023-10-30T12:19:52.805 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: }, 2023-10-30T12:19:52.805 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: "task_status": {} 2023-10-30T12:19:52.805 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: } 2023-10-30T12:19:52.805 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: } 2023-10-30T12:19:52.806 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: } 2023-10-30T12:19:52.806 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: } 2023-10-30T12:19:52.806 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: } 2023-10-30T12:19:52.806 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: 2023-10-30T12:19:52.806 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: attempt 1 of 20 2023-10-30T12:19:52.807 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10899 ms) 2023-10-30T12:19:52.807 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (37014 ms total) 2023-10-30T12:19:52.807 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: 2023-10-30T12:19:52.807 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [----------] Global test environment tear-down 2023-10-30T12:19:52.807 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (37015 ms total) 2023-10-30T12:19:52.807 INFO:tasks.workunit.client.0.smithi148.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-10-30T12:19:52.899 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:19:52.890+0000 10a5b640 -1 osd.0 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:19:54.047 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:19:54.010+0000 10a5b640 -1 osd.0 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:19:54.441 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:19:54.373+0000 1025a640 -1 osd.2 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:1 28.5s0 28:bfdec9d9:test-rados-api-smithi148-92441-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:19:55.312 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:19:55.246+0000 10a5b640 -1 osd.0 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:19:56.412 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:19:56.385+0000 1025a640 -1 osd.2 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:1 28.5s0 28:bfdec9d9:test-rados-api-smithi148-92441-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:19:56.437 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:19:56.417+0000 10a5b640 -1 osd.0 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:19:57.455 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:19:57.383+0000 10a5b640 -1 osd.0 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:19:57.480 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:19:57.440+0000 1025a640 -1 osd.2 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:1 28.5s0 28:bfdec9d9:test-rados-api-smithi148-92441-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:19:58.484 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:19:58.478+0000 1025a640 -1 osd.2 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:1 28.5s0 28:bfdec9d9:test-rados-api-smithi148-92441-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:19:58.567 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:19:58.536+0000 10a5b640 -1 osd.0 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:19:59.609 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:19:59.572+0000 10a5b640 -1 osd.0 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:00.145 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:20:00.143+0000 1025a640 -1 osd.2 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:1 28.5s0 28:bfdec9d9:test-rados-api-smithi148-92441-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:00.847 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:00.772+0000 10a5b640 -1 osd.0 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:00.956 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:03:16.029 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:00.962 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:03:16.036 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:01.017 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:03:16.091 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:01.020 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:03:16.094 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:01.026 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:03:16.074 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:01.036 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:03:16.083 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:01.094 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:03:16.167 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:01.128 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:03:16.176 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:01.145 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:03:16.219 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:01.308 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:20:01.298+0000 1025a640 -1 osd.2 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:1 28.5s0 28:bfdec9d9:test-rados-api-smithi148-92441-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:01.870 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:01.776+0000 10a5b640 -1 osd.0 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:02.328 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:20:02.325+0000 1025a640 -1 osd.2 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:1 28.5s0 28:bfdec9d9:test-rados-api-smithi148-92441-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:03.020 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:02.989+0000 10a5b640 -1 osd.0 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:03.379 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:20:03.359+0000 1025a640 -1 osd.2 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:1 28.5s0 28:bfdec9d9:test-rados-api-smithi148-92441-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:04.588 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:20:04.552+0000 1025a640 -1 osd.2 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:1 28.5s0 28:bfdec9d9:test-rados-api-smithi148-92441-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:04.879 INFO:tasks.workunit.client.0.smithi148.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-10-30T12:20:04.879 INFO:tasks.workunit.client.0.smithi148.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-10-30T12:20:04.879 INFO:tasks.workunit.client.0.smithi148.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-10-30T12:20:04.880 INFO:tasks.workunit.client.0.smithi148.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-10-30T12:20:04.880 INFO:tasks.workunit.client.0.smithi148.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (49070 ms) 2023-10-30T12:20:04.880 INFO:tasks.workunit.client.0.smithi148.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (49071 ms total) 2023-10-30T12:20:04.880 INFO:tasks.workunit.client.0.smithi148.stdout: api_cls_remote_reads: 2023-10-30T12:20:04.881 INFO:tasks.workunit.client.0.smithi148.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-10-30T12:20:04.881 INFO:tasks.workunit.client.0.smithi148.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (49071 ms total) 2023-10-30T12:20:04.881 INFO:tasks.workunit.client.0.smithi148.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-10-30T12:20:05.013 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:05.009+0000 10a5b640 -1 osd.0 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:05.594 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:20:05.586+0000 1025a640 -1 osd.2 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:1 28.5s0 28:bfdec9d9:test-rados-api-smithi148-92441-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:05.975 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: Running main() from gmock_main.cc 2023-10-30T12:20:05.975 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-10-30T12:20:05.975 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [----------] Global test environment set-up. 2023-10-30T12:20:05.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-10-30T12:20:05.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-10-30T12:20:05.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ OK ] LibRadosPools.PoolList (4278 ms) 2023-10-30T12:20:05.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-10-30T12:20:05.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (5409 ms) 2023-10-30T12:20:05.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-10-30T12:20:05.977 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3971 ms) 2023-10-30T12:20:05.977 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-10-30T12:20:05.977 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4422 ms) 2023-10-30T12:20:05.977 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-10-30T12:20:05.977 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (5433 ms) 2023-10-30T12:20:05.978 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-10-30T12:20:05.978 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6983 ms) 2023-10-30T12:20:05.978 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-10-30T12:20:05.978 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5615 ms) 2023-10-30T12:20:05.978 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-10-30T12:20:05.979 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5241 ms) 2023-10-30T12:20:05.979 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-10-30T12:20:05.979 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8995 ms) 2023-10-30T12:20:05.979 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [----------] 9 tests from LibRadosPools (50348 ms total) 2023-10-30T12:20:05.979 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: 2023-10-30T12:20:05.980 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [----------] Global test environment tear-down 2023-10-30T12:20:05.980 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (50349 ms total) 2023-10-30T12:20:05.980 INFO:tasks.workunit.client.0.smithi148.stdout: api_pool: [ PASSED ] 9 tests. 2023-10-30T12:20:07.195 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:07.112+0000 10a5b640 -1 osd.0 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:08.339 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:08.335+0000 10a5b640 -1 osd.0 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:09.373 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:09.362+0000 10a5b640 -1 osd.0 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:10.357 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:10.355+0000 10a5b640 -1 osd.0 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:11.349 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:11.346+0000 10a5b640 -1 osd.0 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:12.353 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:12.347+0000 10a5b640 -1 osd.0 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:13.653 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:13.589+0000 10a5b640 -1 osd.0 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:14.615 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:14.594+0000 10a5b640 -1 osd.0 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:16.077 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:15.837+0000 10a5b640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:16.932 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:16.903+0000 10a5b640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:18.243 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:18.153+0000 10a5b640 -1 osd.0 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:19.296 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:19.176+0000 10a5b640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:20.169 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:20.160+0000 10a5b640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:21.242 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:21.221+0000 10a5b640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:22.253 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:22.232+0000 10a5b640 -1 osd.0 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:23.426 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:20:23.258+0000 10a5b640 -1 osd.0 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e28) 2023-10-30T12:20:34.556 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: Running main() from gmock_main.cc 2023-10-30T12:20:34.557 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-10-30T12:20:34.557 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [----------] Global test environment set-up. 2023-10-30T12:20:34.557 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-10-30T12:20:34.557 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-10-30T12:20:34.558 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (947 ms) 2023-10-30T12:20:34.558 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-10-30T12:20:34.558 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (213 ms) 2023-10-30T12:20:34.558 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-10-30T12:20:34.558 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (69421 ms) 2023-10-30T12:20:34.559 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-10-30T12:20:34.559 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:20.162018+0000 osd.0 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'test-rados-api-smithi148-92464-1' : 1 ]) 2023-10-30T12:20:34.559 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:20.661585+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 8239.952ms) (OSD_SLOW_PING_TIME_BACK) 2023-10-30T12:20:34.559 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:20.666455+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 7936.336ms) (OSD_SLOW_PING_TIME_FRONT) 2023-10-30T12:20:34.560 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:20.667089+0000 mon.a [WRN] Health check failed: Reduced data availability: 9 pgs inactive (PG_AVAILABILITY) 2023-10-30T12:20:34.560 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:21.056210+0000 mon.a [WRN] Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T12:20:34.560 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:21.223307+0000 osd.0 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'test-rados-api-smithi148-92464-1' : 1 ]) 2023-10-30T12:20:34.560 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:22.233342+0000 osd.0 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'test-rados-api-smithi148-92464-1' : 1 ]) 2023-10-30T12:20:34.560 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:23.259799+0000 osd.0 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'test-rados-api-smithi148-92464-1' : 1 ]) 2023-10-30T12:20:34.560 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:22.971714+0000 mon.c [INF] from='client.? 172.21.15.148:0/833259277' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi148-92193-4"}]: dispatch 2023-10-30T12:20:34.561 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:24.579676+0000 mon.a [WRN] Health check update: 1 slow ops, oldest one blocked for 58 sec, osd.0 has slow ops (SLOW_OPS) 2023-10-30T12:20:34.561 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:24.583941+0000 mon.b [INF] from='client.? 172.21.15.148:0/1011796717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi148-92016-10"}]: dispatch 2023-10-30T12:20:34.561 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:24.950750+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi148-92193-4"}]: dispatch 2023-10-30T12:20:34.561 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:25.281229+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi148-92198-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:20:34.561 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:27.457411+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi148-92016-10"}]: dispatch 2023-10-30T12:20:34.562 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:27.891312+0000 mon.a [WRN] Health check update: Reduced data availability: 11 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2023-10-30T12:20:34.562 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:28.516669+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi148-92193-4"}]': finished 2023-10-30T12:20:34.562 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:28.868860+0000 mon.c [INF] from='client.? 172.21.15.148:0/17841433' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-10-30T12:20:37.323 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-10-30T12:20:37.323 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-10-30T12:20:37.323 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-10-30T12:20:37.323 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-10-30T12:20:37.324 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-10-30T12:20:37.324 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (685 ms) 2023-10-30T12:20:37.324 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-10-30T12:20:37.324 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (147 ms) 2023-10-30T12:20:37.324 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-10-30T12:20:37.324 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (80753 ms) 2023-10-30T12:20:37.325 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (81585 ms total) 2023-10-30T12:20:37.325 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: 2023-10-30T12:20:37.325 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-10-30T12:20:37.325 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (81585 ms total) 2023-10-30T12:20:37.325 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-10-30T12:20:43.614 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:03:58.688 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:43.635 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:03:58.706 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:44.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: Running main() from gmock_main.cc 2023-10-30T12:20:44.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-10-30T12:20:44.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-10-30T12:20:44.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-10-30T12:20:44.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-10-30T12:20:44.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-10-30T12:20:44.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-10-30T12:20:44.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (19983 ms) 2023-10-30T12:20:44.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-10-30T12:20:44.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (4611 ms) 2023-10-30T12:20:44.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-10-30T12:20:44.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (36855 ms) 2023-10-30T12:20:44.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-10-30T12:20:44.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (4061 ms) 2023-10-30T12:20:44.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-10-30T12:20:44.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (190 ms) 2023-10-30T12:20:44.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-10-30T12:20:44.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (209 ms) 2023-10-30T12:20:44.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-10-30T12:20:44.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (1525 ms) 2023-10-30T12:20:44.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-10-30T12:20:44.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (1746 ms) 2023-10-30T12:20:44.118 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-10-30T12:20:44.118 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (2074 ms) 2023-10-30T12:20:44.118 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-10-30T12:20:44.118 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (855 ms) 2023-10-30T12:20:44.118 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-10-30T12:20:44.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (150 ms) 2023-10-30T12:20:44.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-10-30T12:20:44.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (136 ms) 2023-10-30T12:20:44.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-10-30T12:20:44.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (2061 ms) 2023-10-30T12:20:44.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-10-30T12:20:44.120 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (365 ms) 2023-10-30T12:20:44.120 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-10-30T12:20:44.120 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (389 ms) 2023-10-30T12:20:44.120 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-10-30T12:20:44.120 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (93 ms) 2023-10-30T12:20:44.121 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (75303 ms total) 2023-10-30T12:20:44.121 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: 2023-10-30T12:20:44.121 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-10-30T12:20:44.121 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (88309 ms total) 2023-10-30T12:20:44.121 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-10-30T12:20:44.146 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:03:59.220 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:44.687 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:03:59.760 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:44.690 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:03:59.764 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:44.772 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-10-30T12:20:44.772 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-10-30T12:20:44.773 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-10-30T12:20:44.773 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-10-30T12:20:44.773 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: seed 92294 2023-10-30T12:20:44.773 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-10-30T12:20:44.773 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (13268 ms) 2023-10-30T12:20:44.774 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-10-30T12:20:44.774 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (18716 ms) 2023-10-30T12:20:44.774 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-10-30T12:20:44.774 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (259 ms) 2023-10-30T12:20:44.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-10-30T12:20:44.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (1986 ms) 2023-10-30T12:20:44.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-10-30T12:20:44.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (13698 ms) 2023-10-30T12:20:44.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (47927 ms total) 2023-10-30T12:20:44.775 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: 2023-10-30T12:20:44.776 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-10-30T12:20:44.776 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-10-30T12:20:44.776 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (7010 ms) 2023-10-30T12:20:44.776 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-10-30T12:20:44.776 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (941 ms) 2023-10-30T12:20:44.777 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-10-30T12:20:44.777 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1407 ms) 2023-10-30T12:20:44.777 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-10-30T12:20:44.777 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (7072 ms) 2023-10-30T12:20:44.777 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (16430 ms total) 2023-10-30T12:20:44.778 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: 2023-10-30T12:20:44.778 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-10-30T12:20:44.778 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (89093 ms total) 2023-10-30T12:20:44.778 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-10-30T12:20:44.824 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:03:59.871 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:44.836 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:03:59.884 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:45.010 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:04:00.084 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:45.023 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:04:00.070 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:45.105 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2023-10-30T12:20:45.106 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [----------] Global test environment set-up. 2023-10-30T12:20:45.106 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [----------] 7 tests from LibRadosList 2023-10-30T12:20:45.106 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-10-30T12:20:45.106 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosList.ListObjects (66182 ms) 2023-10-30T12:20:45.106 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-10-30T12:20:45.107 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (845 ms) 2023-10-30T12:20:45.107 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-10-30T12:20:45.107 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: myset foo1,foo2,foo3 2023-10-30T12:20:45.107 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo1 2023-10-30T12:20:45.107 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo2 2023-10-30T12:20:45.108 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo3 2023-10-30T12:20:45.108 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: myset foo1,foo4,foo5 2023-10-30T12:20:45.108 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo4 2023-10-30T12:20:45.108 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo5 2023-10-30T12:20:45.108 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo1 2023-10-30T12:20:45.108 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: myset foo6,foo7 2023-10-30T12:20:45.109 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo7 2023-10-30T12:20:45.109 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo6 2023-10-30T12:20:45.109 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-10-30T12:20:45.109 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: ns1:foo4 2023-10-30T12:20:45.109 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: ns1:foo5 2023-10-30T12:20:45.109 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: ns2:foo7 2023-10-30T12:20:45.109 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: ns2:foo6 2023-10-30T12:20:45.110 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: ns1:foo1 2023-10-30T12:20:45.110 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: :foo1 2023-10-30T12:20:45.110 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: :foo2 2023-10-30T12:20:45.110 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: :foo3 2023-10-30T12:20:45.110 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (9709 ms) 2023-10-30T12:20:45.111 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-10-30T12:20:45.111 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 1 0 2023-10-30T12:20:45.111 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 10 0 2023-10-30T12:20:45.111 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 13 0 2023-10-30T12:20:45.111 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 7 0 2023-10-30T12:20:45.111 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 14 0 2023-10-30T12:20:45.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 0 0 2023-10-30T12:20:45.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 15 0 2023-10-30T12:20:45.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 11 0 2023-10-30T12:20:45.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 5 0 2023-10-30T12:20:45.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 8 0 2023-10-30T12:20:45.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 6 0 2023-10-30T12:20:45.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 3 0 2023-10-30T12:20:45.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 4 0 2023-10-30T12:20:45.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 12 0 2023-10-30T12:20:45.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 9 0 2023-10-30T12:20:45.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 2 0 2023-10-30T12:20:45.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-10-30T12:20:45.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (3852 ms) 2023-10-30T12:20:45.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-10-30T12:20:45.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: x cursor=MIN 2023-10-30T12:20:45.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=1 cursor=7:02547ec2:::1:head 2023-10-30T12:20:45.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=10 cursor=7:52ea6a34:::10:head 2023-10-30T12:20:45.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=13 cursor=7:566253c9:::13:head 2023-10-30T12:20:45.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=7 cursor=7:5c6b0b28:::7:head 2023-10-30T12:20:45.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=14 cursor=7:62a1935d:::14:head 2023-10-30T12:20:45.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=0 cursor=7:6cac518f:::0:head 2023-10-30T12:20:45.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=15 cursor=7:863748b0:::15:head 2023-10-30T12:20:45.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=11 cursor=7:89d3ae78:::11:head 2023-10-30T12:20:45.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=5 cursor=7:b29083e3:::5:head 2023-10-30T12:20:45.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=8 cursor=7:bd63b0f1:::8:head 2023-10-30T12:20:45.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=6 cursor=7:c4fdafeb:::6:head 2023-10-30T12:20:45.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=3 cursor=7:cfc208b3:::3:head 2023-10-30T12:20:45.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=4 cursor=7:d83876eb:::4:head 2023-10-30T12:20:45.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=12 cursor=7:de5d7c5f:::12:head 2023-10-30T12:20:45.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=9 cursor=7:e960b815:::9:head 2023-10-30T12:20:45.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > oid=2 cursor=7:f905c69b:::2:head 2023-10-30T12:20:45.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: FIRST> seek to MIN oid=1 2023-10-30T12:20:45.118 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=1 cursor=7:02547ec2:::1:head 2023-10-30T12:20:45.118 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:02547ec2:::1:head 2023-10-30T12:20:45.118 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:02547ec2:::1:head -> 1 2023-10-30T12:20:45.118 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=10 cursor=7:52ea6a34:::10:head 2023-10-30T12:20:45.118 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:52ea6a34:::10:head 2023-10-30T12:20:45.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2023-10-30T12:20:45.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=13 cursor=7:566253c9:::13:head 2023-10-30T12:20:45.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:566253c9:::13:head 2023-10-30T12:20:45.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:566253c9:::13:head -> 13 2023-10-30T12:20:45.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=7 cursor=7:5c6b0b28:::7:head 2023-10-30T12:20:45.119 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2023-10-30T12:20:45.120 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2023-10-30T12:20:45.690 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=14 cursor=7:62a1935d:::14:h api_cmd: got: 2023-10-30T12:20:29.002782+0000 mon.c [INF] from='client.? 172.21.15.148:0/1840415149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi148-91959-16"}]: dispatch 2023-10-30T12:20:45.690 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:30.053253+0000 client.admin [INF] onexx 2023-10-30T12:20:45.691 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:30.601670+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-10-30T12:20:45.691 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:30.631723+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi148-91959-16"}]: dispatch 2023-10-30T12:20:45.691 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:33.232013+0000 mon.c [INF] from='client.? 172.21.15.148:0/1840415149' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi148-91959-16"}]: dispatch 2023-10-30T12:20:45.691 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:33.335475+0000 mon.b [INF] from='client.? 172.21.15.148:0/1011796717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi148-92016-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:20:45.691 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:33.640821+0000 mon.c [INF] from='client.? 172.21.15.148:0/3234176010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi148-92346-7"}]: dispatch 2023-10-30T12:20:45.692 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:34.100787+0000 mon.a [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2023-10-30T12:20:45.692 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:34.612874+0000 mon.c [INF] from='client.? 172.21.15.148:0/17841433' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-10-30T12:20:45.692 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:34.661927+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi148-91959-16"}]: dispatch 2023-10-30T12:20:45.692 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:34.674363+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi148-92346-7"}]: dispatch 2023-10-30T12:20:45.692 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:35.017406+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi148-92016-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:20:45.692 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:35.029680+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi148-92193-4", "tierpool": "test-rados-api-smithi148-92193-4-cache"}]': finished 2023-10-30T12:20:45.693 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:35.031584+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi148-92441-1"}]': finished 2023-10-30T12:20:45.693 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:37.118232+0000 client.admin [INF] twoxx 2023-10-30T12:20:45.693 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:37.131228+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-10-30T12:20:45.693 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:36.854853+0000 mon.b [INF] from='client.? 172.21.15.148:0/2447747211' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi148-92441-1"}]: dispatch 2023-10-30T12:20:45.693 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:36.857662+0000 mon.b [INF] from='client.? 172.21.15.148:0/763362389' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi148-91970-22"}]: dispatch 2023-10-30T12:20:48.339 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:37.017217+0000 mon.c [INF] from='client.? 172.21.15.148:0/1840415149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi148-91959-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:20:48.339 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:37.060313+0000 mon.c [INF] from='client.? 172.21.15.148:0/3485765053' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi148-91936-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:20:48.340 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:38.255556+0000 mon.a [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T12:20:48.340 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:38.356982+0000 mon.c [INF] from='client.? 172.21.15.148:0/2199812940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi148-92409-12"}]: dispatch 2023-10-30T12:20:48.340 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:38.710610+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi148-91959-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:20:48.340 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:38.718217+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi148-91936-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:20:48.340 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:38.795750+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi148-92441-1"}]: dispatch 2023-10-30T12:20:48.341 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:38.807042+0000 mon.a [INF] from='client.? 172.21.15.148:0/1441565245' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi148-92294-7"}]': finished 2023-10-30T12:20:48.341 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:38.811772+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi148-92346-7"}]': finished 2023-10-30T12:20:48.341 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:38.814506+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi148-92016-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T12:20:48.341 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:38.971454+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi148-91970-22"}]: dispatch 2023-10-30T12:20:48.341 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:39.779477+0000 mon.a [INF] from='client.? 172.21.15.148:0/1441565245' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi148-92294-7"}]: dispatch 2023-10-30T12:20:48.342 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:39.912401+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi148-92409-12"}]: dispatch 2023-10-30T12:20:48.342 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:40.358826+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-10-30T12:20:48.342 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:40.040214+0000 mon.c [INF] from='client.? 172.21.15.148:0/3234176010' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi148-92346-7"}]: dispatch 2023-10-30T12:20:48.342 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:40.118882+0000 mon.b [INF] from='client.? 172.21.15.148:0/1011796717' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi148-92016-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi148-92016-10"}]: dispatch 2023-10-30T12:20:48.342 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:40.123975+0000 mon.b [INF] from='client.? 172.21.15.148:0/763362389' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi148-91970-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:20:49.339 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-1ead 2023-10-30T12:20:49.339 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:62a1935d:::14:head 2023-10-30T12:20:49.339 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:62a1935d:::14:head -> 14 2023-10-30T12:20:49.339 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=0 cursor=7:6cac518f:::0:head 2023-10-30T12:20:49.339 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:6cac518f:::0:head 2023-10-30T12:20:49.340 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:6cac518f:::0:head -> 0 2023-10-30T12:20:49.340 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=15 cursor=7:863748b0:::15:head 2023-10-30T12:20:49.340 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:863748b0:::15:head 2023-10-30T12:20:49.340 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:863748b0:::15:head -> 15 2023-10-30T12:20:49.340 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=11 cursor=7:89d3ae78:::11:head 2023-10-30T12:20:49.341 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:89d3ae78:::11:head 2023-10-30T12:20:49.341 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2023-10-30T12:20:49.341 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=5 cursor=7:b29083e3:::5:head 2023-10-30T12:20:49.341 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:b29083e3:::5:head 2023-10-30T12:20:49.341 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:b29083e3:::5:head -> 5 2023-10-30T12:20:49.342 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=8 cursor=7:bd63b0f1:::8:head 2023-10-30T12:20:49.342 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2023-10-30T12:20:49.342 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2023-10-30T12:20:49.342 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=6 cursor=7:c4fdafeb:::6:head 2023-10-30T12:20:49.342 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2023-10-30T12:20:49.343 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2023-10-30T12:20:49.343 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=3 cursor=7:cfc208b3:::3:head 2023-10-30T12:20:49.343 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:cfc208b3:::3:head 2023-10-30T12:20:49.343 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2023-10-30T12:20:49.343 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=4 cursor=7:d83876eb:::4:head 2023-10-30T12:20:49.343 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:d83876eb:::4:head 2023-10-30T12:20:49.344 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:d83876eb:::4:head -> 4 2023-10-30T12:20:49.344 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=12 cursor=7:de5d7c5f:::12:head 2023-10-30T12:20:49.344 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2023-10-30T12:20:49.344 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2023-10-30T12:20:49.344 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=9 cursor=7:e960b815:::9:head 2023-10-30T12:20:49.345 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:e960b815:::9:head 2023-10-30T12:20:49.345 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:e960b815:::9:head -> 9 2023-10-30T12:20:49.345 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : oid=2 cursor=7:f905c69b:::2:head 2023-10-30T12:20:49.345 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:f905c69b:::2:head 2023-10-30T12:20:49.345 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:f905c69b:::2:head -> 2 2023-10-30T12:20:49.345 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2023-10-30T12:20:49.346 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:5c6b0b28:::7:head expected=7:5c6b0b28:::7:head 2023-10-30T12:20:49.346 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2023-10-30T12:20:49.346 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=7 expected=7 2023-10-30T12:20:49.346 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:cfc208b3:::3:head 2023-10-30T12:20:49.346 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:cfc208b3:::3:head expected=7:cfc208b3:::3:head 2023-10-30T12:20:49.347 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2023-10-30T12:20:49.347 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=3 expected=3 2023-10-30T12:20:49.347 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:02547ec2:::1:head 2023-10-30T12:20:49.347 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:02547ec2:::1:head expected=7:02547ec2:::1:head 2023-10-30T12:20:49.347 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:02547ec2:::1:head -> 1 2023-10-30T12:20:49.348 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=1 expected=1 2023-10-30T12:20:49.348 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:52ea6a34:::10:head 2023-10-30T12:20:49.348 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:52ea6a34:::10:head expected=7:52ea6a34:::10:head 2023-10-30T12:20:49.348 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2023-10-30T12:20:49.348 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=10 expected=10 2023-10-30T12:20:49.348 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:89d3ae78:::11:head 2023-10-30T12:20:49.349 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:89d3ae78:::11:head expected=7:89d3ae78:::11:head 2023-10-30T12:20:49.349 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2023-10-30T12:20:49.349 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=11 expected=11 2023-10-30T12:20:49.349 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:863748b0:::15:head 2023-10-30T12:20:49.349 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:863748b0:::15:head expected=7:863748b0:::15:head 2023-10-30T12:20:49.350 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:863748b0:::15:head -> 15 2023-10-30T12:20:49.350 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=15 expected=15 2023-10-30T12:20:49.350 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:566253c9:::13:head 2023-10-30T12:20:49.350 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:566253c9:::13:head expected=7:566253c9:::13:head 2023-10-30T12:20:49.350 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:566253c9:::13:head -> 13 2023-10-30T12:20:49.351 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=13 expected=13 2023-10-30T12:20:49.351 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2023-10-30T12:20:49.351 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:de5d7c5f:::12:head expected=7:de5d7c5f:::12:head 2023-10-30T12:20:49.351 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2023-10-30T12:20:49.351 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=12 expected=12 2023-10-30T12:20:49.352 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:d83876eb:::4:head 2023-10-30T12:20:50.387 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: Running main() from gmock_main.cc 2023-10-30T12:20:50.387 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-10-30T12:20:50.387 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [----------] Global test environment set-up. 2023-10-30T12:20:50.387 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-10-30T12:20:50.388 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-10-30T12:20:50.388 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ OK ] LibRadosStat.Stat (21621 ms) 2023-10-30T12:20:50.388 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-10-30T12:20:50.388 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (6587 ms) 2023-10-30T12:20:50.388 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-10-30T12:20:50.388 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ OK ] LibRadosStat.StatNS (21269 ms) 2023-10-30T12:20:50.389 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-10-30T12:20:50.389 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (5 ms) 2023-10-30T12:20:50.389 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-10-30T12:20:50.389 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (1120 ms) 2023-10-30T12:20:50.389 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [----------] 5 tests from LibRadosStat (50602 ms total) 2023-10-30T12:20:50.390 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: 2023-10-30T12:20:50.390 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-10-30T12:20:50.390 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-10-30T12:20:50.390 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (5368 ms) 2023-10-30T12:20:50.390 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-10-30T12:20:50.391 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (3105 ms) 2023-10-30T12:20:50.391 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-10-30T12:20:50.391 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (2917 ms) 2023-10-30T12:20:50.391 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-10-30T12:20:50.391 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1590 ms) 2023-10-30T12:20:50.392 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (12980 ms total) 2023-10-30T12:20:50.392 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: 2023-10-30T12:20:50.392 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [----------] Global test environment tear-down 2023-10-30T12:20:50.392 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (94696 ms total) 2023-10-30T12:20:50.392 INFO:tasks.workunit.client.0.smithi148.stdout: api_stat: [ PASSED ] 9 tests. 2023-10-30T12:20:51.852 INFO:tasks.workunit.client.0.smithi148.stdout:0-30T12:20:40.693550+0000 mon.b [INF] from='client.? 172.21.15.148:0/3807816281' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi148-91920-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:20:51.852 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:40.696028+0000 mon.b [INF] from='client.? 172.21.15.148:0/1424586608' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi148-92575-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:20:51.853 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:41.852333+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi148-92016-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi148-92016-10"}]: dispatch 2023-10-30T12:20:51.853 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:41.874740+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi148-91970-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:20:51.853 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:41.881337+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi148-91920-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:20:51.853 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:41.885798+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi148-92346-7"}]: dispatch 2023-10-30T12:20:51.853 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:41.891297+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi148-92575-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:20:51.854 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:41.912470+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi148-91959-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T12:20:51.854 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:41.916904+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi148-91936-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:20:51.854 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:41.920061+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi148-92441-1"}]': finished 2023-10-30T12:20:51.854 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:41.922205+0000 mon.a [INF] from='client.? 172.21.15.148:0/1441565245' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi148-92294-7"}]': finished 2023-10-30T12:20:51.855 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:43.482302+0000 mon.c [INF] from='client.? 172.21.15.148:0/2199812940' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi148-92409-12"}]: dispatch 2023-10-30T12:20:51.855 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:43.617138+0000 mon.c [INF] from='client.? 172.21.15.148:0/17841433' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-10-30T12:20:51.855 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:43.681068+0000 mon.c [INF] from='client.? 172.21.15.148:0/1840415149' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi148-91959-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi148-91959-16"}]: dispatch 2023-10-30T12:20:51.855 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:43.951855+0000 mon.b [INF] from='client.? 172.21.15.148:0/531701059' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi148-92011-10"}]: dispatch 2023-10-30T12:20:51.855 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:44.624329+0000 mon.a [WRN] Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T12:20:58.997 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd api_lock: Running main() from gmock_main.cc 2023-10-30T12:20:58.998 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-10-30T12:20:58.998 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [----------] Global test environment set-up. 2023-10-30T12:20:58.998 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-10-30T12:20:58.999 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-10-30T12:20:58.999 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (15375 ms) 2023-10-30T12:20:58.999 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-10-30T12:20:58.999 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLock.LockShared (7539 ms) 2023-10-30T12:20:58.999 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-10-30T12:20:58.999 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (7381 ms) 2023-10-30T12:20:59.000 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-10-30T12:20:59.000 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (4196 ms) 2023-10-30T12:20:59.000 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-10-30T12:20:59.000 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (6097 ms) 2023-10-30T12:20:59.000 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-10-30T12:20:59.001 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLock.Unlock (1277 ms) 2023-10-30T12:20:59.001 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-10-30T12:20:59.001 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (3982 ms) 2023-10-30T12:20:59.001 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-10-30T12:20:59.001 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (4040 ms) 2023-10-30T12:20:59.002 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [----------] 8 tests from LibRadosLock (49888 ms total) 2023-10-30T12:20:59.002 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: 2023-10-30T12:20:59.002 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-10-30T12:20:59.002 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-10-30T12:20:59.002 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (6115 ms) 2023-10-30T12:20:59.003 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-10-30T12:20:59.003 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (6774 ms) 2023-10-30T12:20:59.003 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-10-30T12:20:59.003 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (2484 ms) 2023-10-30T12:20:59.003 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-10-30T12:20:59.004 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1486 ms) 2023-10-30T12:20:59.004 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-10-30T12:20:59.004 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (2452 ms) 2023-10-30T12:20:59.004 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-10-30T12:20:59.005 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (444 ms) 2023-10-30T12:20:59.005 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-10-30T12:20:59.005 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (436 ms) 2023-10-30T12:20:59.005 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-10-30T12:20:59.006 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (1286 ms) 2023-10-30T12:20:59.006 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (21477 ms total) 2023-10-30T12:20:59.006 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: 2023-10-30T12:20:59.006 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [----------] Global test environment tear-down 2023-10-30T12:20:59.006 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (103455 ms total) 2023-10-30T12:20:59.007 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock: [ PASSED ] 16 tests. 2023-10-30T12:21:00.562 INFO:tasks.workunit.client.0.smithi148.stdout:: got: 2023-10-30T12:20:45.484297+0000 client.admin [INF] threexx 2023-10-30T12:21:00.562 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:45.549534+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi148-92409-12"}]: dispatch 2023-10-30T12:21:00.563 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:45.562281+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-10-30T12:21:00.563 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:45.569727+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi148-91959-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi148-91959-16"}]: dispatch 2023-10-30T12:21:00.563 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:45.673090+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi148-92011-10"}]: dispatch 2023-10-30T12:21:00.563 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:45.699584+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi148-91970-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T12:21:00.563 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:45.702793+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi148-91920-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:21:00.564 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:45.705610+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi148-92346-7"}]': finished 2023-10-30T12:21:00.564 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: got: 2023-10-30T12:20:45.707780+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi148-92575-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:21:00.564 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (34237 ms) 2023-10-30T12:21:00.564 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (104818 ms total) 2023-10-30T12:21:00.564 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: 2023-10-30T12:21:00.565 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [----------] Global test environment tear-down 2023-10-30T12:21:00.565 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (104818 ms total) 2023-10-30T12:21:00.565 INFO:tasks.workunit.client.0.smithi148.stdout: api_cmd: [ PASSED ] 4 tests. 2023-10-30T12:21:12.394 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_1_[92725]: starting. 2023-10-30T12:21:12.395 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_1_[92725]: creating pool ceph_test_rados_list_parallel.smithi148-92684 2023-10-30T12:21:12.395 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_1_[92725]: created object 0... 2023-10-30T12:21:12.396 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_1_[92725]: created object 25... 2023-10-30T12:21:12.396 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_1_[92725]: created object 49... 2023-10-30T12:21:12.396 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_1_[92725]: finishing. 2023-10-30T12:21:12.397 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_1_[92725]: shutting down. 2023-10-30T12:21:12.397 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_2_[92726]: starting. 2023-10-30T12:21:12.397 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_2_[92726]: listing objects. 2023-10-30T12:21:12.397 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_2_[92726]: listed object 0... 2023-10-30T12:21:12.398 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_2_[92726]: listed object 25... 2023-10-30T12:21:12.398 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_2_[92726]: saw 50 objects 2023-10-30T12:21:12.399 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_2_[92726]: shutting down. 2023-10-30T12:21:12.399 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.400 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_3_[114162]: starting. 2023-10-30T12:21:12.400 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_3_[114162]: creating pool ceph_test_rados_list_parallel.smithi148-92684 2023-10-30T12:21:12.401 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_3_[114162]: created object 0... 2023-10-30T12:21:12.401 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_3_[114162]: created object 25... 2023-10-30T12:21:12.401 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_3_[114162]: created object 49... 2023-10-30T12:21:12.401 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_3_[114162]: finishing. 2023-10-30T12:21:12.401 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_3_[114162]: shutting down. 2023-10-30T12:21:12.402 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.402 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_4_[114163]: starting. 2023-10-30T12:21:12.402 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_4_[114163]: listing objects. 2023-10-30T12:21:12.402 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_4_[114163]: listed object 0... 2023-10-30T12:21:12.402 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_4_[114163]: listed object 25... 2023-10-30T12:21:12.403 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_4_[114163]: saw 45 objects 2023-10-30T12:21:12.403 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_4_[114163]: shutting down. 2023-10-30T12:21:12.403 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.403 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_5_[114164]: starting. 2023-10-30T12:21:12.403 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_5_[114164]: removed 25 objects... 2023-10-30T12:21:12.404 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_5_[114164]: removed half of the objects 2023-10-30T12:21:12.404 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_5_[114164]: removed 50 objects... 2023-10-30T12:21:12.404 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_5_[114164]: removed 50 objects 2023-10-30T12:21:12.404 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_5_[114164]: shutting down. 2023-10-30T12:21:12.404 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.405 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.405 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_6_[124651]: starting. 2023-10-30T12:21:12.405 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_6_[124651]: creating pool ceph_test_rados_list_parallel.smithi148-92684 2023-10-30T12:21:12.405 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_6_[124651]: created object 0... 2023-10-30T12:21:12.405 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_6_[124651]: created object 25... 2023-10-30T12:21:12.405 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_6_[124651]: created object 49... 2023-10-30T12:21:12.406 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_6_[124651]: finishing. 2023-10-30T12:21:12.406 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_6_[124651]: shutting down. 2023-10-30T12:21:12.406 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.406 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.406 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_7_[124652]: starting. 2023-10-30T12:21:12.407 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_7_[124652]: listing objects. 2023-10-30T12:21:12.407 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_7_[124652]: listed object 0... 2023-10-30T12:21:12.407 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_7_[124652]: listed object 25... 2023-10-30T12:21:12.407 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_7_[124652]: listed object 50... 2023-10-30T12:21:12.407 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_7_[124652]: saw 54 objects 2023-10-30T12:21:12.408 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_7_[124652]: shutting down. 2023-10-30T12:21:12.408 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.408 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.408 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_8_[124653]: starting. 2023-10-30T12:21:12.408 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_8_[124653]: added 25 objects... 2023-10-30T12:21:12.408 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_8_[124653]: added half of the objects 2023-10-30T12:21:12.409 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_8_[124653]: added 50 objects... 2023-10-30T12:21:12.409 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_8_[124653]: added 50 objects 2023-10-30T12:21:12.409 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_8_[124653]: shutting down. 2023-10-30T12:21:12.409 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.409 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:12.469 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: Running main() from gmock_main.cc 2023-10-30T12:21:12.470 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-10-30T12:21:12.470 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [----------] Global test environment set-up. 2023-10-30T12:21:12.470 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-10-30T12:21:12.471 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: seed 91970 2023-10-30T12:21:12.471 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-10-30T12:21:12.471 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-10-30T12:21:12.471 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-10-30T12:21:12.472 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (7022 ms) 2023-10-30T12:21:12.472 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-10-30T12:21:12.472 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (13008 ms) 2023-10-30T12:21:12.472 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-10-30T12:21:12.472 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (810 ms) 2023-10-30T12:21:12.473 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-10-30T12:21:12.473 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (1154 ms) 2023-10-30T12:21:12.473 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-10-30T12:21:12.473 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (496 ms) 2023-10-30T12:21:12.473 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-10-30T12:21:12.474 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2039 ms) 2023-10-30T12:21:12.474 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-10-30T12:21:12.474 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2224 ms) 2023-10-30T12:21:12.474 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-10-30T12:21:12.475 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (3120 ms) 2023-10-30T12:21:12.475 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-10-30T12:21:12.475 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (4790 ms) 2023-10-30T12:21:12.475 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-10-30T12:21:12.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (7329 ms) 2023-10-30T12:21:12.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-10-30T12:21:12.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (14080 ms) 2023-10-30T12:21:12.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-10-30T12:21:12.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (1076 ms) 2023-10-30T12:21:12.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-10-30T12:21:12.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (604 ms) 2023-10-30T12:21:12.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-10-30T12:21:12.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (761 ms) 2023-10-30T12:21:12.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-10-30T12:21:12.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (1269 ms) 2023-10-30T12:21:12.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-10-30T12:21:12.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (1716 ms) 2023-10-30T12:21:12.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-10-30T12:21:12.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (686 ms) 2023-10-30T12:21:12.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-10-30T12:21:12.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (250 ms) 2023-10-30T12:21:12.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-10-30T12:21:12.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (231 ms) 2023-10-30T12:21:12.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (62665 ms total) 2023-10-30T12:21:12.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: 2023-10-30T12:21:12.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-10-30T12:21:12.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-10-30T12:21:12.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (12696 ms) 2023-10-30T12:21:12.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-10-30T12:21:12.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (392 ms) 2023-10-30T12:21:12.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-10-30T12:21:12.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (144 ms) 2023-10-30T12:21:12.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-10-30T12:21:12.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (149 ms) 2023-10-30T12:21:12.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-10-30T12:21:16.573 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.Rou delete_pools_parallel: process_1_[92735]: starting. 2023-10-30T12:21:16.573 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_1_[92735]: creating pool ceph_test_rados_delete_pools_parallel.smithi148-92705 2023-10-30T12:21:16.573 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_1_[92735]: created object 0... 2023-10-30T12:21:16.573 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_1_[92735]: created object 25... 2023-10-30T12:21:16.574 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_1_[92735]: created object 49... 2023-10-30T12:21:16.574 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_1_[92735]: finishing. 2023-10-30T12:21:16.574 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_1_[92735]: shutting down. 2023-10-30T12:21:16.574 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_2_[92736]: starting. 2023-10-30T12:21:16.574 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_2_[92736]: deleting pool ceph_test_rados_delete_pools_parallel.smithi148-92705 2023-10-30T12:21:16.575 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_2_[92736]: shutting down. 2023-10-30T12:21:16.575 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:21:16.575 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_3_[116440]: starting. 2023-10-30T12:21:16.575 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_3_[116440]: creating pool ceph_test_rados_delete_pools_parallel.smithi148-92705 2023-10-30T12:21:16.575 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_3_[116440]: created object 0... 2023-10-30T12:21:16.576 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_3_[116440]: created object 25... 2023-10-30T12:21:16.576 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_3_[116440]: created object 49... 2023-10-30T12:21:16.576 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_3_[116440]: finishing. 2023-10-30T12:21:16.576 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_3_[116440]: shutting down. 2023-10-30T12:21:16.576 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:21:16.577 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_5_[116442]: starting. 2023-10-30T12:21:16.577 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_5_[116442]: listing objects. 2023-10-30T12:21:16.577 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_5_[116442]: listed object 0... 2023-10-30T12:21:16.577 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_5_[116442]: listed object 25... 2023-10-30T12:21:16.577 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_5_[116442]: saw 50 objects 2023-10-30T12:21:16.578 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_5_[116442]: shutting down. 2023-10-30T12:21:16.578 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:21:16.578 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_4_[116441]: starting. 2023-10-30T12:21:16.578 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_4_[116441]: deleting pool ceph_test_rados_delete_pools_parallel.smithi148-92705 2023-10-30T12:21:16.579 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: process_4_[116441]: shutting down. 2023-10-30T12:21:16.579 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:21:16.579 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:21:16.579 INFO:tasks.workunit.client.0.smithi148.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-10-30T12:21:17.085 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_1_[92722]: starting. 2023-10-30T12:21:17.086 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_1_[92722]: creating pool ceph_test_rados_open_pools_parallel.smithi148-92656 2023-10-30T12:21:17.086 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_1_[92722]: created object 0... 2023-10-30T12:21:17.086 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_1_[92722]: created object 25... 2023-10-30T12:21:17.087 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_1_[92722]: created object 49... 2023-10-30T12:21:17.087 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_1_[92722]: finishing. 2023-10-30T12:21:17.087 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_1_[92722]: shutting down. 2023-10-30T12:21:17.087 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_2_[92724]: starting. 2023-10-30T12:21:17.087 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_2_[92724]: rados_pool_create. 2023-10-30T12:21:17.087 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_2_[92724]: rados_ioctx_create. 2023-10-30T12:21:17.088 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_2_[92724]: shutting down. 2023-10-30T12:21:17.088 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: ******************************* 2023-10-30T12:21:17.088 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_3_[119274]: starting. 2023-10-30T12:21:17.088 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_3_[119274]: creating pool ceph_test_rados_open_pools_parallel.smithi148-92656 2023-10-30T12:21:17.088 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_3_[119274]: created object 0... 2023-10-30T12:21:17.089 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_3_[119274]: created object 25... 2023-10-30T12:21:17.089 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_3_[119274]: created object 49... 2023-10-30T12:21:17.089 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_3_[119274]: finishing. 2023-10-30T12:21:17.089 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_3_[119274]: shutting down. 2023-10-30T12:21:17.089 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: ******************************* 2023-10-30T12:21:17.089 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_4_[119276]: starting. 2023-10-30T12:21:17.090 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_4_[119276]: rados_pool_create. 2023-10-30T12:21:17.090 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_4_[119276]: rados_ioctx_create. 2023-10-30T12:21:17.090 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: process_4_[119276]: shutting down. 2023-10-30T12:21:17.090 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: ******************************* 2023-10-30T12:21:17.091 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: ******************************* 2023-10-30T12:21:17.091 INFO:tasks.workunit.client.0.smithi148.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-10-30T12:21:26.889 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-10-30T12:21:26.889 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-10-30T12:21:26.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-10-30T12:21:26.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-10-30T12:21:26.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: seed 92016 2023-10-30T12:21:26.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-10-30T12:21:26.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (22485 ms) 2023-10-30T12:21:26.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-10-30T12:21:26.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (11648 ms) 2023-10-30T12:21:26.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-10-30T12:21:26.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (4666 ms) 2023-10-30T12:21:26.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-10-30T12:21:26.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (4714 ms) 2023-10-30T12:21:26.892 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-10-30T12:21:26.892 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (3195 ms) 2023-10-30T12:21:26.892 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-10-30T12:21:26.892 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (3740 ms) 2023-10-30T12:21:26.892 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-10-30T12:21:26.893 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4322 ms) 2023-10-30T12:21:26.893 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-10-30T12:21:26.893 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3324 ms) 2023-10-30T12:21:26.893 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (58094 ms total) 2023-10-30T12:21:26.893 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: 2023-10-30T12:21:26.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-10-30T12:21:26.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-10-30T12:21:26.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (5983 ms) 2023-10-30T12:21:26.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-10-30T12:21:26.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (104 ms) 2023-10-30T12:21:26.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-10-30T12:21:26.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1463 ms) 2023-10-30T12:21:26.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-10-30T12:21:26.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1498 ms) 2023-10-30T12:21:26.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-10-30T12:21:26.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (284 ms) 2023-10-30T12:21:26.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-10-30T12:21:26.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (249 ms) 2023-10-30T12:21:26.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-10-30T12:21:26.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (292 ms) 2023-10-30T12:21:26.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-10-30T12:21:26.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (150 ms) 2023-10-30T12:21:26.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (10023 ms total) 2023-10-30T12:21:26.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: 2023-10-30T12:21:26.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-10-30T12:21:26.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (131332 ms total) 2023-10-30T12:21:26.898 INFO:tasks.workunit.client.0.smithi148.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-10-30T12:21:26.916 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2023-10-30T12:21:26.917 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [----------] Global test environment set-up. 2023-10-30T12:21:26.917 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-10-30T12:21:26.917 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-10-30T12:21:26.917 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-10-30T12:21:26.918 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-10-30T12:21:26.918 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: 2023-10-30T12:21:26.918 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-10-30T12:21:26.918 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-10-30T12:21:26.918 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2023-10-30T12:19:15.917+0000 7f849e445780 -1 failed for service _ceph-mon._tcp 2023-10-30T12:21:26.919 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: ceph-mon 2023-10-30T12:21:26.919 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: 2023-10-30T12:19:15.917+0000 7f849e445780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-10-30T12:21:26.919 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (306 ms) 2023-10-30T12:21:26.919 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-10-30T12:21:26.919 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2009 ms) 2023-10-30T12:21:26.919 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2315 ms total) 2023-10-30T12:21:26.920 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: 2023-10-30T12:21:26.920 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-10-30T12:21:26.920 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-10-30T12:21:26.920 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097de8f900 2023-10-30T12:21:26.920 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097df62570 2023-10-30T12:21:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097d496b10 2023-10-30T12:21:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097d455f20 2023-10-30T12:21:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097d5d1480 2023-10-30T12:21:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097d5d1e90 2023-10-30T12:21:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097d5d28a0 2023-10-30T12:21:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097d5d32b0 2023-10-30T12:21:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097d5d3cc0 2023-10-30T12:21:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097d5d46d0 2023-10-30T12:21:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 0x56097d5c7d70 2023-10-30T12:21:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: started 11 aios 2023-10-30T12:21:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097de8f900 2023-10-30T12:21:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097df62570 2023-10-30T12:21:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097d496b10 2023-10-30T12:21:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097d455f20 2023-10-30T12:21:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097d5d1480 2023-10-30T12:21:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097d5d1e90 2023-10-30T12:21:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097d5d28a0 2023-10-30T12:21:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097d5d32b0 2023-10-30T12:21:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097d5d3cc0 2023-10-30T12:21:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097d5d46d0 2023-10-30T12:21:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: waiting 0x56097d5c7d70 2023-10-30T12:21:26.925 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: done. 2023-10-30T12:21:26.925 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (18769 ms) 2023-10-30T12:21:26.925 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (18769 ms total) 2023-10-30T12:21:26.925 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: 2023-10-30T12:21:26.925 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-10-30T12:21:26.926 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-10-30T12:21:26.926 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-10-30T12:21:26.926 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-10-30T12:21:26.926 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMisc.Exec (21587 ms) 2023-10-30T12:21:26.926 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-10-30T12:21:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (4325 ms) 2023-10-30T12:21:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-10-30T12:21:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2125 ms) 2023-10-30T12:21:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-10-30T12:21:26.927 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5866 ms) 2023-10-30T12:21:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-10-30T12:21:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-10-30T12:21:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-10-30T12:21:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-10-30T12:21:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-10-30T12:21:26.928 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (71849 ms) 2023-10-30T12:21:26.929 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [----------] 8 tests from LibRadosMisc (105752 ms total) 2023-10-30T12:21:26.929 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: 2023-10-30T12:21:26.929 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [----------] Global test environment tear-down 2023-10-30T12:21:26.930 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (131304 ms total) 2023-10-30T12:21:26.930 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc: [ PASSED ] 12 tests. 2023-10-30T12:21:26.942 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: Running main() from gmock_main.cc 2023-10-30T12:21:26.942 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-10-30T12:21:26.942 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [----------] Global test environment set-up. 2023-10-30T12:21:26.943 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-10-30T12:21:26.943 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-10-30T12:21:26.943 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (15682 ms) 2023-10-30T12:21:26.943 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-10-30T12:21:26.943 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-10-30T12:21:26.944 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-10-30T12:21:26.944 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: no timeout :/ 2023-10-30T12:21:26.944 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: no timeout :/ 2023-10-30T12:21:26.944 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: no timeout :/ 2023-10-30T12:21:26.944 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: no timeout :/ 2023-10-30T12:21:26.945 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: no timeout :/ 2023-10-30T12:21:26.945 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (985 ms) 2023-10-30T12:21:26.945 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-10-30T12:21:26.945 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (33348 ms) 2023-10-30T12:21:26.945 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-10-30T12:21:26.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.Checksum (627 ms) 2023-10-30T12:21:26.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-10-30T12:21:26.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (2203 ms) 2023-10-30T12:21:26.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-10-30T12:21:26.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (678 ms) 2023-10-30T12:21:26.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-10-30T12:21:26.947 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (6410 ms) 2023-10-30T12:21:26.947 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-10-30T12:21:26.947 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (50 ms) 2023-10-30T12:21:26.947 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-10-30T12:21:26.947 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.TruncTest (1133 ms) 2023-10-30T12:21:26.948 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-10-30T12:21:26.948 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (638 ms) 2023-10-30T12:21:26.948 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-10-30T12:21:26.948 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (123 ms) 2023-10-30T12:21:26.948 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-10-30T12:21:26.949 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.RmXattr (842 ms) 2023-10-30T12:21:26.949 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-10-30T12:21:26.949 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIo.XattrIter (339 ms) 2023-10-30T12:21:26.949 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [----------] 14 tests from LibRadosIo (63059 ms total) 2023-10-30T12:21:26.949 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: 2023-10-30T12:21:26.950 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-10-30T12:21:26.950 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-10-30T12:21:26.950 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (5552 ms) 2023-10-30T12:21:26.950 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-10-30T12:21:26.950 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (427 ms) 2023-10-30T12:21:26.951 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-10-30T12:21:26.951 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (321 ms) 2023-10-30T12:21:26.951 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-10-30T12:21:26.951 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (475 ms) 2023-10-30T12:21:26.951 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-10-30T12:21:26.951 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (347 ms) 2023-10-30T12:21:26.952 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-10-30T12:21:26.952 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (219 ms) 2023-10-30T12:21:26.952 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-10-30T12:21:26.952 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (241 ms) 2023-10-30T12:21:26.952 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-10-30T12:21:26.953 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (223 ms) 2023-10-30T12:21:26.953 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-10-30T12:21:26.953 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (2082 ms) 2023-10-30T12:21:26.953 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-10-30T12:21:26.953 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (207 ms) 2023-10-30T12:21:26.953 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [----------] 10 tests from LibRadosIoEC (10094 ms total) 2023-10-30T12:21:26.954 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: 2023-10-30T12:21:26.954 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [----------] Global test environment tear-down 2023-10-30T12:21:26.955 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [==========] 24 tests from 2 test suites ran. (131425 ms total) 2023-10-30T12:21:26.955 INFO:tasks.workunit.client.0.smithi148.stdout: api_io: [ PASSED ] 24 tests. 2023-10-30T12:21:27.000 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:04:42.074 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:27.009 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:04:42.083 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:27.016 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:04:42.063 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:27.020 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:04:42.068 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:27.034 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:04:42.108 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:27.037 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:04:42.111 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:27.144 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:04:42.192 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:27.175 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:04:42.249 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:27.203 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:04:42.277 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:28.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-10-30T12:21:28.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-10-30T12:21:28.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-10-30T12:21:28.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-10-30T12:21:28.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-10-30T12:21:28.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T12:21:28.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (19239 ms) 2023-10-30T12:21:28.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-10-30T12:21:28.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-10-30T12:21:28.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94278699420160 err -107 2023-10-30T12:21:28.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (13481 ms) 2023-10-30T12:21:28.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-10-30T12:21:28.485 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-10-30T12:21:28.486 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94278699420160 err -107 2023-10-30T12:21:28.486 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (9200 ms) 2023-10-30T12:21:28.486 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-10-30T12:21:28.486 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_cb from 4425 notify_id 253403070464 cookie 94278700257984 2023-10-30T12:21:28.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (2517 ms) 2023-10-30T12:21:28.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-10-30T12:21:28.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_cb from 4425 notify_id 296352743424 cookie 94278700257984 2023-10-30T12:21:28.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (15195 ms) 2023-10-30T12:21:28.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-10-30T12:21:28.487 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_cb from 4425 notify_id 317827579904 cookie 94278701876656 2023-10-30T12:21:28.488 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (451 ms) 2023-10-30T12:21:28.488 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-10-30T12:21:28.488 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_cb from 4425 notify_id 317827579904 cookie 94278701876656 2023-10-30T12:21:28.488 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_cb from 4425 notify_id 317827579904 cookie 94278701926608 2023-10-30T12:21:28.488 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (364 ms) 2023-10-30T12:21:28.489 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-10-30T12:21:28.489 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_cb from 4425 notify_id 322122547201 cookie 94278701926608 2023-10-30T12:21:28.489 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_cb from 4425 notify_id 322122547202 cookie 94278701926608 2023-10-30T12:21:28.489 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (5130 ms) 2023-10-30T12:21:28.489 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-10-30T12:21:28.490 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-10-30T12:21:28.490 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94278701926608 err -107 2023-10-30T12:21:28.490 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_cb from 4425 notify_id 335007449093 cookie 94278701912400 2023-10-30T12:21:28.490 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (7351 ms) 2023-10-30T12:21:28.490 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-10-30T12:21:28.491 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94278701912400 err -107 2023-10-30T12:21:28.491 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-10-30T12:21:28.491 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (642 ms) 2023-10-30T12:21:28.491 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (73571 ms total) 2023-10-30T12:21:28.491 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: 2023-10-30T12:21:28.492 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-10-30T12:21:28.492 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-10-30T12:21:28.492 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T12:21:28.492 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (9157 ms) 2023-10-30T12:21:28.492 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (9157 ms total) 2023-10-30T12:21:28.493 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: 2023-10-30T12:21:28.493 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-10-30T12:21:28.493 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (132769 ms total) 2023-10-30T12:21:28.494 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-10-30T12:21:29.110 INFO:tasks.workunit.client.0.smithi148.stdout:ndTripPP2 (100 ms) 2023-10-30T12:21:29.111 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-10-30T12:21:29.111 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (428 ms) 2023-10-30T12:21:29.111 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-10-30T12:21:29.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (2179 ms) 2023-10-30T12:21:29.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-10-30T12:21:29.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (210 ms) 2023-10-30T12:21:29.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-10-30T12:21:29.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (353 ms) 2023-10-30T12:21:29.112 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-10-30T12:21:29.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (247 ms) 2023-10-30T12:21:29.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-10-30T12:21:29.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (137 ms) 2023-10-30T12:21:29.113 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-10-30T12:21:29.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (2185 ms) 2023-10-30T12:21:29.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-10-30T12:21:29.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (1792 ms) 2023-10-30T12:21:29.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-10-30T12:21:29.114 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (475 ms) 2023-10-30T12:21:29.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-10-30T12:21:29.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (384 ms) 2023-10-30T12:21:29.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-10-30T12:21:29.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (107 ms) 2023-10-30T12:21:29.115 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-10-30T12:21:29.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (171 ms) 2023-10-30T12:21:29.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (22149 ms total) 2023-10-30T12:21:29.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: 2023-10-30T12:21:29.116 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [----------] Global test environment tear-down 2023-10-30T12:21:29.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (133589 ms total) 2023-10-30T12:21:29.117 INFO:tasks.workunit.client.0.smithi148.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-10-30T12:21:43.841 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-10-30T12:21:43.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-10-30T12:21:43.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-10-30T12:21:43.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-10-30T12:21:43.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-10-30T12:21:43.842 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: notify 2023-10-30T12:21:43.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (59607 ms) 2023-10-30T12:21:43.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-10-30T12:21:43.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (2040 ms) 2023-10-30T12:21:43.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (61647 ms total) 2023-10-30T12:21:43.843 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: 2023-10-30T12:21:43.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-10-30T12:21:43.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-10-30T12:21:43.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: notify 2023-10-30T12:21:43.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (491 ms) 2023-10-30T12:21:43.844 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-10-30T12:21:43.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: notify 2023-10-30T12:21:43.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (25433 ms) 2023-10-30T12:21:43.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-10-30T12:21:43.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (682 ms) 2023-10-30T12:21:43.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-10-30T12:21:43.845 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (1201 ms) 2023-10-30T12:21:43.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-10-30T12:21:43.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: handle_notify cookie 94574379719168 notify_id 433791696897 notifier_gid 10481 2023-10-30T12:21:43.846 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (1089 ms) 2023-10-30T12:21:43.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-10-30T12:21:43.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: handle_notify cookie 94574379719168 notify_id 442381631490 notifier_gid 10481 2023-10-30T12:21:43.847 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (1080 ms) 2023-10-30T12:21:43.848 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-10-30T12:21:43.848 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: handle_notify cookie 94574379719168 notify_id 446676598785 notifier_gid 10481 2023-10-30T12:21:43.848 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (1172 ms) 2023-10-30T12:21:43.849 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-10-30T12:21:43.849 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: handle_notify cookie 94574381394608 notify_id 450971566082 notifier_gid 10481 2023-10-30T12:21:43.849 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (879 ms) 2023-10-30T12:21:43.850 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-10-30T12:21:43.850 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: handle_notify cookie 94574382014784 notify_id 450971566083 notifier_gid 10481 2023-10-30T12:21:43.850 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (271 ms) 2023-10-30T12:21:43.850 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-10-30T12:21:43.851 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: handle_notify cookie 94574382014784 notify_id 463856467970 notifier_gid 10481 2023-10-30T12:21:43.851 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (2883 ms) 2023-10-30T12:21:43.851 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-10-30T12:21:43.852 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: trying... 2023-10-30T12:21:43.852 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: handle_notify cookie 94574382014784 notify_id 472446402563 notifier_gid 10481 2023-10-30T12:21:43.852 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: timed out 2023-10-30T12:21:44.936 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_p list_parallel: ******************************* 2023-10-30T12:21:44.936 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_9_[138493]: starting. 2023-10-30T12:21:44.936 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_9_[138493]: creating pool ceph_test_rados_list_parallel.smithi148-92684 2023-10-30T12:21:44.937 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_9_[138493]: created object 0... 2023-10-30T12:21:44.937 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_9_[138493]: created object 25... 2023-10-30T12:21:44.937 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_9_[138493]: created object 49... 2023-10-30T12:21:44.937 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_9_[138493]: finishing. 2023-10-30T12:21:44.937 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_9_[138493]: shutting down. 2023-10-30T12:21:44.937 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.938 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.938 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.938 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_10_[138494]: starting. 2023-10-30T12:21:44.938 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_10_[138494]: listing objects. 2023-10-30T12:21:44.938 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_10_[138494]: listed object 0... 2023-10-30T12:21:44.939 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_10_[138494]: listed object 25... 2023-10-30T12:21:44.939 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_10_[138494]: listed object 50... 2023-10-30T12:21:44.939 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_10_[138494]: listed object 75... 2023-10-30T12:21:44.939 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_10_[138494]: saw 99 objects 2023-10-30T12:21:44.939 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_10_[138494]: shutting down. 2023-10-30T12:21:44.939 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.940 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.940 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.940 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_11_[138495]: starting. 2023-10-30T12:21:44.940 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_11_[138495]: added 25 objects... 2023-10-30T12:21:44.940 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_11_[138495]: added half of the objects 2023-10-30T12:21:44.940 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_11_[138495]: added 50 objects... 2023-10-30T12:21:44.941 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_11_[138495]: added 50 objects 2023-10-30T12:21:44.941 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_11_[138495]: shutting down. 2023-10-30T12:21:44.941 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.941 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.941 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.942 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_12_[138496]: starting. 2023-10-30T12:21:44.942 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_12_[138496]: added 25 objects... 2023-10-30T12:21:44.942 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_12_[138496]: added half of the objects 2023-10-30T12:21:44.942 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_12_[138496]: added 50 objects... 2023-10-30T12:21:44.942 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_12_[138496]: added 50 objects 2023-10-30T12:21:44.943 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_12_[138496]: shutting down. 2023-10-30T12:21:44.943 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.943 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.943 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.943 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_13_[138497]: starting. 2023-10-30T12:21:44.943 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_13_[138497]: removed 25 objects... 2023-10-30T12:21:44.944 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_13_[138497]: removed half of the objects 2023-10-30T12:21:44.944 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_13_[138497]: removed 50 objects... 2023-10-30T12:21:44.944 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_13_[138497]: removed 50 objects 2023-10-30T12:21:44.944 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_13_[138497]: shutting down. 2023-10-30T12:21:44.944 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.944 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.945 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.945 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.945 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_14_[151831]: starting. 2023-10-30T12:21:44.945 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_14_[151831]: creating pool ceph_test_rados_list_parallel.smithi148-92684 2023-10-30T12:21:44.945 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_14_[151831]: created object 0... 2023-10-30T12:21:44.945 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_14_[151831]: created object 25... 2023-10-30T12:21:44.946 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_14_[151831]: created object 49... 2023-10-30T12:21:44.946 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_14_[151831]: finishing. 2023-10-30T12:21:44.946 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_14_[151831]: shutting down. 2023-10-30T12:21:44.946 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.946 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.947 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.947 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:44.947 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_15_[151832]: starting. 2023-10-30T12:21:44.947 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_15_[151832]: listing objects. 2023-10-30T12:21:53.044 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_15_[151832]: listed object 0... 2023-10-30T12:21:53.044 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_15_[151832]: listed object 25... 2023-10-30T12:21:53.044 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_15_[151832]: listed object 50... 2023-10-30T12:21:53.045 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_15_[151832]: listed object 75... 2023-10-30T12:21:53.045 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_15_[151832]: listed object 100... 2023-10-30T12:21:53.045 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_15_[151832]: listed object 125... 2023-10-30T12:21:53.045 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_15_[151832]: saw 150 objects 2023-10-30T12:21:53.045 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_15_[151832]: shutting down. 2023-10-30T12:21:53.045 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:53.046 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:53.046 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:53.046 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:53.046 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_16_[151833]: starting. 2023-10-30T12:21:53.046 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_16_[151833]: added 25 objects... 2023-10-30T12:21:53.047 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_16_[151833]: added half of the objects 2023-10-30T12:21:53.047 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_16_[151833]: added 50 objects... 2023-10-30T12:21:53.047 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_16_[151833]: added 50 objects 2023-10-30T12:21:53.047 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: process_16_[151833]: shutting down. 2023-10-30T12:21:53.047 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:53.048 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:53.048 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:53.048 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:53.048 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******************************* 2023-10-30T12:21:53.049 INFO:tasks.workunit.client.0.smithi148.stdout: list_parallel: ******* SUCCESS ********** 2023-10-30T12:22:02.174 INFO:tasks.workunit.client.0.smithi148.stdout:p: flushing 2023-10-30T12:22:02.175 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: flushed 2023-10-30T12:22:02.175 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (4363 ms) 2023-10-30T12:22:02.175 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-10-30T12:22:02.175 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: trying... 2023-10-30T12:22:02.175 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: handle_notify cookie 94574382189040 notify_id 489626271748 notifier_gid 10481 2023-10-30T12:22:02.176 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: timed out 2023-10-30T12:22:02.176 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: flushing 2023-10-30T12:22:02.176 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: flushed 2023-10-30T12:22:02.176 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (4711 ms) 2023-10-30T12:22:02.177 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-10-30T12:22:02.177 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: List watches 2023-10-30T12:22:02.177 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: notify2 2023-10-30T12:22:02.177 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: handle_notify cookie 94574382191184 notify_id 506806140932 notifier_gid 10481 2023-10-30T12:22:02.177 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: notify2 done 2023-10-30T12:22:02.178 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: watch_check 2023-10-30T12:22:02.178 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: unwatch2 2023-10-30T12:22:02.178 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: flushing 2023-10-30T12:22:02.178 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: done 2023-10-30T12:22:02.178 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (4729 ms) 2023-10-30T12:22:02.179 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-10-30T12:22:02.179 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: List watches 2023-10-30T12:22:02.179 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: notify2 2023-10-30T12:22:02.179 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: handle_notify cookie 94574382191184 notify_id 528280977411 notifier_gid 10481 2023-10-30T12:22:02.179 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: notify2 done 2023-10-30T12:22:02.180 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: watch_check 2023-10-30T12:22:02.180 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: unwatch2 2023-10-30T12:22:02.180 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: flushing 2023-10-30T12:22:02.180 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: done 2023-10-30T12:22:02.180 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3704 ms) 2023-10-30T12:22:02.181 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (52688 ms total) 2023-10-30T12:22:02.181 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: 2023-10-30T12:22:02.181 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-10-30T12:22:02.181 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (166439 ms total) 2023-10-30T12:22:02.182 INFO:tasks.workunit.client.0.smithi148.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-10-30T12:22:07.229 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:05:22.302 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:07.233 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:05:22.307 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:07.301 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:05:22.348 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:07.307 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:05:22.354 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:07.319 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:05:22.393 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:07.323 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:05:22.395 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:07.380 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:05:22.454 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:07.436 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:05:22.484 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:07.467 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:05:22.541 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:16.475 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: Running main() from gmock_main.cc 2023-10-30T12:22:16.475 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-10-30T12:22:16.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [----------] Global test environment set-up. 2023-10-30T12:22:16.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-10-30T12:22:16.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-10-30T12:22:16.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (10395 ms) 2023-10-30T12:22:16.476 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-10-30T12:22:16.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (11640 ms) 2023-10-30T12:22:16.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-10-30T12:22:16.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (28171 ms) 2023-10-30T12:22:16.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-10-30T12:22:16.477 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (4339 ms) 2023-10-30T12:22:16.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (54545 ms total) 2023-10-30T12:22:16.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: 2023-10-30T12:22:16.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-10-30T12:22:16.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-10-30T12:22:16.478 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (35924 ms) 2023-10-30T12:22:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-10-30T12:22:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (25875 ms) 2023-10-30T12:22:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-10-30T12:22:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (6468 ms) 2023-10-30T12:22:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (68267 ms total) 2023-10-30T12:22:16.479 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: 2023-10-30T12:22:16.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-10-30T12:22:16.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-10-30T12:22:16.480 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3568 ms) 2023-10-30T12:22:16.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-10-30T12:22:16.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (3595 ms) 2023-10-30T12:22:16.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-10-30T12:22:16.481 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (4350 ms) 2023-10-30T12:22:16.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-10-30T12:22:16.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (3240 ms) 2023-10-30T12:22:16.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (14754 ms total) 2023-10-30T12:22:16.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: 2023-10-30T12:22:16.482 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-10-30T12:22:16.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-10-30T12:22:16.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (6955 ms) 2023-10-30T12:22:16.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-10-30T12:22:16.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4767 ms) 2023-10-30T12:22:16.483 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (11722 ms total) 2023-10-30T12:22:16.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: 2023-10-30T12:22:16.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [----------] Global test environment tear-down 2023-10-30T12:22:16.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (180834 ms total) 2023-10-30T12:22:16.484 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-10-30T12:22:27.210 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:05:42.284 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:27.273 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:05:42.347 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:27.336 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:05:42.410 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:27.337 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:05:42.384 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:27.339 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:05:42.413 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:27.340 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:05:42.388 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:27.387 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:05:42.461 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:27.401 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:05:42.449 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:27.418 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:05:42.492 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:38.621 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-10-30T12:22:38.621 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-10-30T12:22:38.621 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-10-30T12:22:38.622 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-10-30T12:22:38.622 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: seed 92236 2023-10-30T12:22:38.622 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-10-30T12:22:38.622 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (22154 ms) 2023-10-30T12:22:38.622 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-10-30T12:22:38.623 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (5226 ms) 2023-10-30T12:22:38.623 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-10-30T12:22:38.623 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (6497 ms) 2023-10-30T12:22:38.623 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-10-30T12:22:38.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (4251 ms) 2023-10-30T12:22:38.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-10-30T12:22:38.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (53208 ms) 2023-10-30T12:22:38.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (91337 ms total) 2023-10-30T12:22:38.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: 2023-10-30T12:22:38.625 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-10-30T12:22:38.625 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-10-30T12:22:38.625 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (12786 ms) 2023-10-30T12:22:38.625 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-10-30T12:22:38.625 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (6773 ms) 2023-10-30T12:22:38.626 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-10-30T12:22:38.626 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (10323 ms) 2023-10-30T12:22:38.626 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-10-30T12:22:38.626 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (5343 ms) 2023-10-30T12:22:38.626 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-10-30T12:22:38.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (3148 ms) 2023-10-30T12:22:38.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-10-30T12:22:38.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6839-gcc36dd85/rpm/el9/BUILD/ceph-18.0.0-6839-gcc36dd85/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-10-30T12:22:38.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: 2023-10-30T12:22:38.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-10-30T12:22:38.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-10-30T12:22:38.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi148-92236-7 2023-10-30T12:22:38.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: waiting for snaps to purge 2023-10-30T12:22:38.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (19119 ms) 2023-10-30T12:22:38.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (57492 ms total) 2023-10-30T12:22:38.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: 2023-10-30T12:22:38.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-10-30T12:22:38.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-10-30T12:22:38.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3142 ms) 2023-10-30T12:22:38.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-10-30T12:22:38.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2325 ms) 2023-10-30T12:22:38.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-10-30T12:22:38.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (3054 ms) 2023-10-30T12:22:38.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-10-30T12:22:38.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1588 ms) 2023-10-30T12:22:38.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (10109 ms total) 2023-10-30T12:22:38.631 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: 2023-10-30T12:22:38.631 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-10-30T12:22:47.279 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:06:02.353 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:47.328 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:06:02.376 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:47.329 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:06:02.377 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:47.335 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:06:02.408 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:47.395 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:06:02.469 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:47.396 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:06:02.443 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:47.461 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:06:02.535 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:47.462 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:06:02.536 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:47.580 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:06:02.654 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:47.884 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] api_list: : cursor()=7:d83876eb:::4:head expected=7:d83876eb:::4:head 2023-10-30T12:22:47.884 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:d83876eb:::4:head -> 4 2023-10-30T12:22:47.885 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=4 expected=4 2023-10-30T12:22:47.885 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2023-10-30T12:22:47.885 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:c4fdafeb:::6:head expected=7:c4fdafeb:::6:head 2023-10-30T12:22:47.885 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2023-10-30T12:22:47.885 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=6 expected=6 2023-10-30T12:22:47.886 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2023-10-30T12:22:47.886 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:bd63b0f1:::8:head expected=7:bd63b0f1:::8:head 2023-10-30T12:22:47.886 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2023-10-30T12:22:47.886 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=8 expected=8 2023-10-30T12:22:47.886 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:f905c69b:::2:head 2023-10-30T12:22:47.887 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:f905c69b:::2:head expected=7:f905c69b:::2:head 2023-10-30T12:22:47.887 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:f905c69b:::2:head -> 2 2023-10-30T12:22:47.887 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=2 expected=2 2023-10-30T12:22:47.887 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:e960b815:::9:head 2023-10-30T12:22:47.887 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:e960b815:::9:head expected=7:e960b815:::9:head 2023-10-30T12:22:47.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:e960b815:::9:head -> 9 2023-10-30T12:22:47.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=9 expected=9 2023-10-30T12:22:47.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:b29083e3:::5:head 2023-10-30T12:22:47.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:b29083e3:::5:head expected=7:b29083e3:::5:head 2023-10-30T12:22:47.888 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:b29083e3:::5:head -> 5 2023-10-30T12:22:47.889 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=5 expected=5 2023-10-30T12:22:47.889 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:6cac518f:::0:head 2023-10-30T12:22:47.889 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:6cac518f:::0:head expected=7:6cac518f:::0:head 2023-10-30T12:22:47.889 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:6cac518f:::0:head -> 0 2023-10-30T12:22:47.889 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=0 expected=0 2023-10-30T12:22:47.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : seek to 7:62a1935d:::14:head 2023-10-30T12:22:47.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : cursor()=7:62a1935d:::14:head expected=7:62a1935d:::14:head 2023-10-30T12:22:47.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: > 7:62a1935d:::14:head -> 14 2023-10-30T12:22:47.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: : entry=14 expected=14 2023-10-30T12:22:47.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (12822 ms) 2023-10-30T12:22:47.890 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-10-30T12:22:47.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (55506 ms) 2023-10-30T12:22:47.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-10-30T12:22:47.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: split 0/5 -> MIN 7:33333333::::head 2023-10-30T12:22:47.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: split 1/5 -> 7:33333333::::head 7:66666666::::head 2023-10-30T12:22:47.891 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: split 2/5 -> 7:66666666::::head 7:99999999::::head 2023-10-30T12:22:47.892 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: split 3/5 -> 7:99999999::::head 7:cccccccc::::head 2023-10-30T12:22:47.892 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: split 4/5 -> 7:cccccccc::::head MAX 2023-10-30T12:22:47.892 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (47907 ms) 2023-10-30T12:22:47.892 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [----------] 7 tests from LibRadosList (196823 ms total) 2023-10-30T12:22:47.892 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 2023-10-30T12:22:47.893 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-10-30T12:22:47.893 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-10-30T12:22:47.893 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2397 ms) 2023-10-30T12:22:47.893 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-10-30T12:22:47.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: myset foo1,foo2,foo3 2023-10-30T12:22:47.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo1 2023-10-30T12:22:47.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo2 2023-10-30T12:22:47.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo3 2023-10-30T12:22:47.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: myset foo1,foo4,foo5 2023-10-30T12:22:47.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo4 2023-10-30T12:22:47.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo5 2023-10-30T12:22:47.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo1 2023-10-30T12:22:47.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: myset foo6,foo7 2023-10-30T12:22:47.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo7 2023-10-30T12:22:47.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: foo6 2023-10-30T12:22:47.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-10-30T12:22:47.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: ns1:foo4 2023-10-30T12:22:47.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: ns1:foo5 2023-10-30T12:22:47.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: ns2:foo7 2023-10-30T12:22:47.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: ns2:foo6 2023-10-30T12:22:47.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: ns1:foo1 2023-10-30T12:22:47.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: :foo1 2023-10-30T12:22:47.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: :foo2 2023-10-30T12:22:47.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: :foo3 2023-10-30T12:22:47.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (3594 ms) 2023-10-30T12:22:47.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-10-30T12:22:47.898 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 1 0 2023-10-30T12:22:47.898 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 10 0 2023-10-30T12:22:47.898 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 13 0 2023-10-30T12:22:47.973 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: Running main() from gmock_main.cc 2023-10-30T12:22:47.974 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-10-30T12:22:47.974 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [----------] Global test environment set-up. 2023-10-30T12:22:47.974 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-10-30T12:22:47.974 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-10-30T12:22:47.975 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4393 ms) 2023-10-30T12:22:47.975 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-10-30T12:22:47.975 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (22891 ms) 2023-10-30T12:22:47.975 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-10-30T12:22:47.975 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (19996 ms) 2023-10-30T12:22:47.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-10-30T12:22:47.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (9625 ms) 2023-10-30T12:22:47.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-10-30T12:22:47.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (17640 ms) 2023-10-30T12:22:47.976 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-10-30T12:22:47.977 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (27313 ms) 2023-10-30T12:22:47.977 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-10-30T12:22:47.977 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (29363 ms) 2023-10-30T12:22:47.977 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-10-30T12:22:47.977 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (7402 ms) 2023-10-30T12:22:47.978 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-10-30T12:22:47.978 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4176 ms) 2023-10-30T12:22:47.978 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-10-30T12:22:47.978 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (6793 ms) 2023-10-30T12:22:47.978 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-10-30T12:22:47.979 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (5665 ms) 2023-10-30T12:22:47.979 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-10-30T12:22:47.979 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (4324 ms) 2023-10-30T12:22:47.979 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-10-30T12:22:47.979 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (4032 ms) 2023-10-30T12:22:47.980 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-10-30T12:22:47.980 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (4125 ms) 2023-10-30T12:22:47.980 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-10-30T12:22:47.980 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.Flush (3146 ms) 2023-10-30T12:22:47.980 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-10-30T12:22:47.980 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3993 ms) 2023-10-30T12:22:47.981 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-10-30T12:22:47.981 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3980 ms) 2023-10-30T12:22:47.981 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-10-30T12:22:47.981 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3157 ms) 2023-10-30T12:22:47.981 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-10-30T12:22:47.982 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (4338 ms) 2023-10-30T12:22:47.982 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-10-30T12:22:47.982 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2454 ms) 2023-10-30T12:22:47.983 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-10-30T12:22:47.983 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (4390 ms) 2023-10-30T12:22:47.983 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-10-30T12:22:47.983 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3702 ms) 2023-10-30T12:22:47.983 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-10-30T12:22:47.984 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3130 ms) 2023-10-30T12:22:47.984 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-10-30T12:22:47.984 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4104 ms) 2023-10-30T12:22:47.984 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-10-30T12:22:47.984 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3811 ms) 2023-10-30T12:22:47.984 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-10-30T12:22:47.985 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (4518 ms) 2023-10-30T12:22:47.985 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [----------] 26 tests from LibRadosAio (212463 ms total) 2023-10-30T12:22:47.985 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: 2023-10-30T12:22:47.985 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-10-30T12:22:51.358 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleW api_aio_pp: Running main() from gmock_main.cc 2023-10-30T12:22:51.358 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-10-30T12:22:51.358 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-10-30T12:22:51.358 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-10-30T12:22:51.359 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-10-30T12:22:51.359 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4436 ms) 2023-10-30T12:22:51.359 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-10-30T12:22:51.359 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (30540 ms) 2023-10-30T12:22:51.359 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-10-30T12:22:51.360 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (23044 ms) 2023-10-30T12:22:51.360 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-10-30T12:22:51.360 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (13521 ms) 2023-10-30T12:22:51.360 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-10-30T12:22:51.360 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (27114 ms) 2023-10-30T12:22:51.361 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-10-30T12:22:51.361 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (31378 ms) 2023-10-30T12:22:51.361 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-10-30T12:22:51.361 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (5583 ms) 2023-10-30T12:22:51.362 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-10-30T12:22:51.362 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (4025 ms) 2023-10-30T12:22:51.362 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-10-30T12:22:51.362 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3162 ms) 2023-10-30T12:22:51.362 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-10-30T12:22:51.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3709 ms) 2023-10-30T12:22:51.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-10-30T12:22:51.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3062 ms) 2023-10-30T12:22:51.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-10-30T12:22:51.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (4588 ms) 2023-10-30T12:22:51.364 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-10-30T12:22:51.364 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (4389 ms) 2023-10-30T12:22:51.364 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-10-30T12:22:51.364 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (5065 ms) 2023-10-30T12:22:51.364 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-10-30T12:22:51.365 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (6232 ms) 2023-10-30T12:22:51.365 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-10-30T12:22:51.365 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2999 ms) 2023-10-30T12:22:51.366 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-10-30T12:22:51.366 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3828 ms) 2023-10-30T12:22:51.366 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-10-30T12:22:51.366 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4281 ms) 2023-10-30T12:22:51.367 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-10-30T12:22:51.367 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3215 ms) 2023-10-30T12:22:51.367 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-10-30T12:22:51.367 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (4634 ms) 2023-10-30T12:22:51.367 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-10-30T12:22:51.368 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (4409 ms) 2023-10-30T12:22:51.368 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-10-30T12:22:51.368 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3340 ms) 2023-10-30T12:22:51.368 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-10-30T12:22:51.368 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3415 ms) 2023-10-30T12:22:51.369 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-10-30T12:22:51.369 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4160 ms) 2023-10-30T12:22:51.369 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-10-30T12:22:51.369 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3817 ms) 2023-10-30T12:22:51.369 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-10-30T12:22:51.370 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (4520 ms) 2023-10-30T12:22:51.370 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-10-30T12:22:56.893 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRado api_list: 7 0 2023-10-30T12:22:56.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 14 0 2023-10-30T12:22:56.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 0 0 2023-10-30T12:22:56.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 15 0 2023-10-30T12:22:56.894 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 11 0 2023-10-30T12:22:56.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 5 0 2023-10-30T12:22:56.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 8 0 2023-10-30T12:22:56.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 6 0 2023-10-30T12:22:56.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 3 0 2023-10-30T12:22:56.895 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 4 0 2023-10-30T12:22:56.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 12 0 2023-10-30T12:22:56.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 9 0 2023-10-30T12:22:56.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 2 0 2023-10-30T12:22:56.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-10-30T12:22:56.896 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (5415 ms) 2023-10-30T12:22:56.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [----------] 3 tests from LibRadosListEC (11406 ms total) 2023-10-30T12:22:56.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 2023-10-30T12:22:56.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-10-30T12:22:56.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-10-30T12:22:56.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3305 ms) 2023-10-30T12:22:56.897 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [----------] 1 test from LibRadosListNP (3305 ms total) 2023-10-30T12:22:56.898 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: 2023-10-30T12:22:56.898 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [----------] Global test environment tear-down 2023-10-30T12:22:56.898 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [==========] 11 tests from 3 test suites ran. (221348 ms total) 2023-10-30T12:22:56.898 INFO:tasks.workunit.client.0.smithi148.stdout: api_list: [ PASSED ] 11 tests. 2023-10-30T12:22:58.513 INFO:tasks.workunit.client.0.smithi148.stdout: LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-10-30T12:22:58.514 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (6611 ms) 2023-10-30T12:22:58.514 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-10-30T12:22:58.514 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (6126 ms) 2023-10-30T12:22:58.514 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-10-30T12:22:58.515 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4314 ms) 2023-10-30T12:22:58.515 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (17051 ms total) 2023-10-30T12:22:58.515 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: 2023-10-30T12:22:58.515 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-10-30T12:22:58.515 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (222861 ms total) 2023-10-30T12:22:58.516 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-10-30T12:22:58.516 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-10-30T12:22:58.516 INFO:tasks.workunit.client.0.smithi148.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-10-30T12:23:12.454 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:06:27.527 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:12.458 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:06:27.532 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:12.513 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:06:27.560 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:12.514 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:06:27.562 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:12.524 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:06:27.598 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:12.530 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:06:27.604 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:12.540 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:06:27.614 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:12.569 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:06:27.616 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:12.583 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:06:27.657 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:32.816 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:06:47.889 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:32.859 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:06:47.932 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:32.878 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:06:47.925 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:32.891 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:06:47.938 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:32.892 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:06:47.966 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:32.896 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:06:47.970 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:32.940 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:06:48.013 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:32.968 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:06:48.041 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:32.984 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:06:48.031 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:35.943 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-10-30T12:23:35.944 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-10-30T12:23:35.944 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-10-30T12:23:35.944 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-10-30T12:23:35.944 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-10-30T12:23:35.945 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-10-30T12:23:35.945 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-10-30T12:23:35.945 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: 2023-10-30T12:23:35.945 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-10-30T12:23:35.945 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-10-30T12:23:35.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (28098 ms) 2023-10-30T12:23:35.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-10-30T12:23:35.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (9888 ms) 2023-10-30T12:23:35.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-10-30T12:23:35.946 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (12197 ms) 2023-10-30T12:23:35.947 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-10-30T12:23:35.947 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (24601 ms) 2023-10-30T12:23:35.947 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-10-30T12:23:35.947 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (26781 ms) 2023-10-30T12:23:35.947 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-10-30T12:23:35.948 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (29382 ms) 2023-10-30T12:23:35.948 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-10-30T12:23:35.948 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (129200 ms) 2023-10-30T12:23:35.948 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (260147 ms total) 2023-10-30T12:23:35.948 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: 2023-10-30T12:23:35.949 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-10-30T12:23:35.949 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (260147 ms total) 2023-10-30T12:23:35.949 INFO:tasks.workunit.client.0.smithi148.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-10-30T12:23:52.855 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:07.928 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:52.915 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:07.989 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:52.970 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:08.043 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:52.986 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:08.060 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:52.988 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:08.036 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:52.994 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:08.068 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:52.996 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:08.043 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:53.052 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:08.099 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:53.056 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:08.129 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:10.037 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:25.110 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:10.091 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:25.164 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:10.134 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:25.208 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:10.145 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:25.219 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:10.147 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:25.220 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:10.157 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:25.204 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:10.160 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:25.207 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:10.190 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:25.263 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:10.214 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:25.261 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:17.846 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:32.919 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:17.848 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:32.921 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:17.869 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:32.943 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:17.892 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:32.939 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:17.894 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:32.941 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:17.899 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:32.973 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:17.901 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:32.974 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:17.913 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:32.960 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:17.922 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:32.996 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:27.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-10-30T12:24:27.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-10-30T12:24:27.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-10-30T12:24:27.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-10-30T12:24:27.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-10-30T12:24:27.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-10-30T12:24:27.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: 2023-10-30T12:24:27.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-10-30T12:24:27.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: seed 92173 2023-10-30T12:24:27.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-10-30T12:24:27.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (7 ms) 2023-10-30T12:24:27.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-10-30T12:24:27.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (30030 ms) 2023-10-30T12:24:27.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-10-30T12:24:27.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (19580 ms) 2023-10-30T12:24:27.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-10-30T12:24:27.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (4774 ms) 2023-10-30T12:24:27.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-10-30T12:24:27.457 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (4637 ms) 2023-10-30T12:24:27.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-10-30T12:24:27.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (43 ms) 2023-10-30T12:24:27.458 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-10-30T12:24:27.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (1683 ms) 2023-10-30T12:24:27.459 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-10-30T12:24:27.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (1251 ms) 2023-10-30T12:24:27.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-10-30T12:24:27.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (57 ms) 2023-10-30T12:24:27.460 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-10-30T12:24:27.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (205 ms) 2023-10-30T12:24:27.461 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-10-30T12:24:27.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (147 ms) 2023-10-30T12:24:27.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-10-30T12:24:27.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (84 ms) 2023-10-30T12:24:27.462 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-10-30T12:24:27.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (3356 ms) 2023-10-30T12:24:27.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-10-30T12:24:27.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: osd_max_attr_size = 0 2023-10-30T12:24:27.463 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-10-30T12:24:27.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (149542 ms) 2023-10-30T12:24:27.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-10-30T12:24:27.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (9655 ms) 2023-10-30T12:24:27.464 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-10-30T12:24:27.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-10-30T12:24:27.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: done waiting, doing copies 2023-10-30T12:24:27.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: waiting for final deep scrubs... 2023-10-30T12:24:27.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: done waiting 2023-10-30T12:24:27.465 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (76796 ms) 2023-10-30T12:24:27.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-10-30T12:24:27.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (231 ms) 2023-10-30T12:24:27.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-10-30T12:24:27.466 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (48 ms) 2023-10-30T12:24:27.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-10-30T12:24:27.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5201 ms) 2023-10-30T12:24:27.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-10-30T12:24:27.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-10-30T12:24:27.467 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-10-30T12:24:27.468 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-10-30T12:24:27.468 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-10-30T12:24:27.468 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-10-30T12:24:38.010 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:53.084 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:38.014 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:53.088 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:38.076 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:53.123 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:38.077 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:53.151 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:38.078 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:53.124 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:38.081 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:53.153 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:38.096 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:07:53.170 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:38.135 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:07:53.182 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:38.143 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:07:53.216 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:53.162 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------rite 2023-10-30T12:24:53.162 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7696 ms) 2023-10-30T12:24:53.162 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-10-30T12:24:53.162 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8300 ms) 2023-10-30T12:24:53.163 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-10-30T12:24:53.163 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (8996 ms) 2023-10-30T12:24:53.163 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-10-30T12:24:53.163 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (9101 ms) 2023-10-30T12:24:53.163 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-10-30T12:24:53.164 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (8173 ms) 2023-10-30T12:24:53.164 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-10-30T12:24:53.164 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6749 ms) 2023-10-30T12:24:53.164 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-10-30T12:24:53.164 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6937 ms) 2023-10-30T12:24:53.165 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-10-30T12:24:53.165 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (8580 ms) 2023-10-30T12:24:53.165 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-10-30T12:24:53.165 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (8799 ms) 2023-10-30T12:24:53.165 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-10-30T12:24:53.165 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6726 ms) 2023-10-30T12:24:53.166 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-10-30T12:24:53.166 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (8047 ms) 2023-10-30T12:24:53.166 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-10-30T12:24:53.166 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7946 ms) 2023-10-30T12:24:53.167 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-10-30T12:24:53.167 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7545 ms) 2023-10-30T12:24:53.167 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-10-30T12:24:53.167 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6851 ms) 2023-10-30T12:24:53.167 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-10-30T12:24:53.168 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6780 ms) 2023-10-30T12:24:53.168 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-10-30T12:24:53.168 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7960 ms) 2023-10-30T12:24:53.168 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (125186 ms total) 2023-10-30T12:24:53.168 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: 2023-10-30T12:24:53.169 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [----------] Global test environment tear-down 2023-10-30T12:24:53.169 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (337650 ms total) 2023-10-30T12:24:53.169 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio: [ PASSED ] 42 tests. 2023-10-30T12:24:58.128 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:08:13.202 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:58.133 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:08:13.207 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:58.178 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:08:13.225 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:58.182 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:08:13.256 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:58.183 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:08:13.230 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:58.187 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:08:13.261 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:58.191 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:08:13.265 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:58.224 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:08:13.271 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:58.232 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:08:13.306 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:59.623 INFO:tasks.workunit.client.0.smithi148.stdout:] 22 tests from LibRadosMiscPP (307328 ms total) 2023-10-30T12:24:59.623 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: 2023-10-30T12:24:59.623 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-10-30T12:24:59.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-10-30T12:24:59.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (5847 ms) 2023-10-30T12:24:59.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (5847 ms total) 2023-10-30T12:24:59.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: 2023-10-30T12:24:59.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-10-30T12:24:59.624 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-10-30T12:24:59.625 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (452 ms) 2023-10-30T12:24:59.625 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-10-30T12:24:59.625 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (73 ms) 2023-10-30T12:24:59.625 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (525 ms total) 2023-10-30T12:24:59.625 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: 2023-10-30T12:24:59.626 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-10-30T12:24:59.626 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-10-30T12:24:59.626 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (1217 ms) 2023-10-30T12:24:59.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-10-30T12:24:59.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (59 ms) 2023-10-30T12:24:59.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (1276 ms total) 2023-10-30T12:24:59.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: 2023-10-30T12:24:59.627 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-10-30T12:24:59.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-10-30T12:24:59.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (367 ms) 2023-10-30T12:24:59.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-10-30T12:24:59.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (84 ms) 2023-10-30T12:24:59.628 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (451 ms total) 2023-10-30T12:24:59.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: 2023-10-30T12:24:59.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-10-30T12:24:59.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-10-30T12:24:59.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1679 ms) 2023-10-30T12:24:59.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1679 ms total) 2023-10-30T12:24:59.629 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: 2023-10-30T12:24:59.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-10-30T12:24:59.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (343983 ms total) 2023-10-30T12:24:59.630 INFO:tasks.workunit.client.0.smithi148.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-10-30T12:25:18.228 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:08:33.301 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:18.235 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:08:33.309 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:18.288 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:08:33.361 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:18.288 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:08:33.335 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:18.291 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:08:33.338 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:18.296 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:08:33.369 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:18.303 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:08:33.377 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:18.332 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:08:33.379 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:18.346 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:08:33.419 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:29.355 INFO:tasks.workunit.client.0.smithi148.stdout:sAio.RoundTripAppendPP (3382 ms) 2023-10-30T12:25:29.356 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-10-30T12:25:29.356 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3263 ms) 2023-10-30T12:25:29.356 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-10-30T12:25:29.356 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3888 ms) 2023-10-30T12:25:29.357 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-10-30T12:25:29.357 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3112 ms) 2023-10-30T12:25:29.357 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-10-30T12:25:29.357 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: setting pool EIO 2023-10-30T12:25:29.357 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: waiting for 46 2023-10-30T12:25:29.358 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: max_success 5, min_failed 6 2023-10-30T12:25:29.358 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (7048 ms) 2023-10-30T12:25:29.358 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-10-30T12:25:29.358 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4264 ms) 2023-10-30T12:25:29.358 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (237424 ms total) 2023-10-30T12:25:29.359 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: 2023-10-30T12:25:29.359 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-10-30T12:25:29.359 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-10-30T12:25:29.359 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3654 ms) 2023-10-30T12:25:29.359 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-10-30T12:25:29.360 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9184 ms) 2023-10-30T12:25:29.360 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-10-30T12:25:29.360 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (16275 ms) 2023-10-30T12:25:29.360 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-10-30T12:25:29.361 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2839 ms) 2023-10-30T12:25:29.361 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (31952 ms total) 2023-10-30T12:25:29.361 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: 2023-10-30T12:25:29.361 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-10-30T12:25:29.361 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-10-30T12:25:29.362 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3400 ms) 2023-10-30T12:25:29.362 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3401 ms total) 2023-10-30T12:25:29.362 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: 2023-10-30T12:25:29.362 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-10-30T12:25:29.362 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-10-30T12:25:29.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14678 ms) 2023-10-30T12:25:29.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-10-30T12:25:29.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7057 ms) 2023-10-30T12:25:29.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-10-30T12:25:29.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7175 ms) 2023-10-30T12:25:29.363 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-10-30T12:25:29.364 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7921 ms) 2023-10-30T12:25:29.364 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-10-30T12:25:29.364 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3382 ms) 2023-10-30T12:25:29.364 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-10-30T12:25:29.364 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7637 ms) 2023-10-30T12:25:29.365 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-10-30T12:25:29.365 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7163 ms) 2023-10-30T12:25:29.365 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-10-30T12:25:29.365 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7989 ms) 2023-10-30T12:25:29.365 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-10-30T12:25:29.366 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7301 ms) 2023-10-30T12:25:29.366 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-10-30T12:25:29.366 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6922 ms) 2023-10-30T12:25:29.366 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-10-30T12:25:29.366 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7648 ms) 2023-10-30T12:25:29.367 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-10-30T12:25:29.367 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (8229 ms) 2023-10-30T12:25:29.367 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-10-30T12:25:29.367 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7967 ms) 2023-10-30T12:25:38.479 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:08:53.553 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:38.555 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:08:53.629 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:38.604 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:08:53.678 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:38.617 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:08:53.664 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:38.619 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:08:53.667 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:38.632 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:08:53.705 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:38.635 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:08:53.709 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:38.675 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:08:53.722 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:38.694 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:08:53.768 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:56.293 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:11.366 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:56.296 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:11.369 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:56.337 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:11.410 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:56.347 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:11.394 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:56.348 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:11.395 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:56.353 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:11.426 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:56.354 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:11.428 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:56.386 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:11.433 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:56.395 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:11.468 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:58.517 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:13.591 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:58.568 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:13.642 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:58.584 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:13.658 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:58.630 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:13.678 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:58.634 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:13.681 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:58.636 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:13.710 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:58.643 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:13.716 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:58.651 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:13.699 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:58.661 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:13.734 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:17.376 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-10-30T12:26:17.376 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3360 ms) 2023-10-30T12:26:17.376 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-10-30T12:26:17.377 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7811 ms) 2023-10-30T12:26:17.377 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-10-30T12:26:17.377 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7092 ms) 2023-10-30T12:26:17.377 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-10-30T12:26:17.377 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (8385 ms) 2023-10-30T12:26:17.378 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-10-30T12:26:17.378 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6961 ms) 2023-10-30T12:26:17.378 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-10-30T12:26:17.378 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6569 ms) 2023-10-30T12:26:17.379 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-10-30T12:26:17.379 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7840 ms) 2023-10-30T12:26:17.379 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (149088 ms total) 2023-10-30T12:26:17.379 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: 2023-10-30T12:26:17.379 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-10-30T12:26:17.379 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (421865 ms total) 2023-10-30T12:26:17.380 INFO:tasks.workunit.client.0.smithi148.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-10-30T12:26:18.507 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:33.580 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:18.512 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:33.585 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:18.547 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:33.620 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:18.559 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:33.606 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:18.561 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:33.635 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:18.563 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:33.610 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:18.565 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:33.638 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:18.599 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:33.646 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:18.601 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:33.674 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:38.631 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:53.704 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:38.695 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:53.769 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:38.699 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:53.746 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:38.703 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:53.750 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:38.728 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:09:53.775 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:38.730 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:09:53.804 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:38.774 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:53.847 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:38.778 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:53.852 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:38.831 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:09:53.904 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:58.619 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:10:13.692 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:58.621 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:10:13.695 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:58.654 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:10:13.727 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:58.667 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:10:13.714 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:58.668 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:10:13.715 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:58.670 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:10:13.744 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:58.675 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:10:13.748 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:58.697 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:10:13.744 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:58.705 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:10:13.778 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:18.890 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:10:33.963 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:18.894 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:10:33.967 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:18.929 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:10:34.003 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:18.936 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:10:33.983 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:18.937 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:10:33.985 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:18.950 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:10:34.024 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:18.951 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:10:34.025 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:18.970 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:10:34.017 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:18.987 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:10:34.061 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:31.986 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-10-30T12:27:31.986 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-10-30T12:27:31.986 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-10-30T12:27:31.987 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: seed 92193 2023-10-30T12:27:31.987 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-10-30T12:27:31.987 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (25575 ms) 2023-10-30T12:27:31.987 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-10-30T12:27:31.987 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (69569 ms) 2023-10-30T12:27:31.987 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-10-30T12:27:31.988 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (623 ms) 2023-10-30T12:27:31.988 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (95767 ms total) 2023-10-30T12:27:31.988 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: 2023-10-30T12:27:31.988 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-10-30T12:27:31.988 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-10-30T12:27:31.988 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (19046 ms) 2023-10-30T12:27:31.989 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-10-30T12:27:31.989 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (16349 ms) 2023-10-30T12:27:31.989 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-10-30T12:27:31.989 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (19387 ms) 2023-10-30T12:27:31.989 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-10-30T12:27:31.990 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: my_snaps [4] 2023-10-30T12:27:31.990 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: my_snaps [5,4] 2023-10-30T12:27:31.990 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: my_snaps [6,5,4] 2023-10-30T12:27:31.990 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-10-30T12:27:31.990 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: promoting some heads 2023-10-30T12:27:31.990 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: promoting from clones for snap 7 2023-10-30T12:27:31.991 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: promoting from clones for snap 6 2023-10-30T12:27:31.991 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: promoting from clones for snap 5 2023-10-30T12:27:31.991 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: promoting from clones for snap 4 2023-10-30T12:27:31.991 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: waiting for scrubs... 2023-10-30T12:27:31.991 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: done waiting 2023-10-30T12:27:31.992 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (146733 ms) 2023-10-30T12:27:31.992 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-10-30T12:27:31.992 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9864 ms) 2023-10-30T12:27:31.992 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-10-30T12:27:31.992 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (10298 ms) 2023-10-30T12:27:31.992 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-10-30T12:27:31.993 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8981 ms) 2023-10-30T12:27:31.993 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-10-30T12:27:31.993 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (9024 ms) 2023-10-30T12:27:31.993 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-10-30T12:27:31.993 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (12556 ms) 2023-10-30T12:27:31.994 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-10-30T12:27:31.994 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9276 ms) 2023-10-30T12:27:31.994 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-10-30T12:27:31.994 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10547 ms) 2023-10-30T12:27:31.994 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-10-30T12:27:31.994 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (38572 ms) 2023-10-30T12:27:31.995 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-10-30T12:27:31.995 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10119 ms) 2023-10-30T12:27:31.995 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-10-30T12:27:31.995 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (9965 ms) 2023-10-30T12:27:31.995 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-10-30T12:27:31.996 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (15374 ms) 2023-10-30T12:27:31.996 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-10-30T12:27:31.996 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8540 ms) 2023-10-30T12:27:31.996 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-10-30T12:27:31.996 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11299 ms) 2023-10-30T12:27:38.949 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:10:54.022 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:38.953 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:10:54.026 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:38.983 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:10:54.056 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:39.000 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:10:54.073 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:39.001 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:10:54.075 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:39.005 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:10:54.053 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:39.007 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:10:54.054 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:39.031 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:10:54.104 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:39.036 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:10:54.083 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:53.697 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-10-30T12:27:53.698 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: hmm, no HitSet yet 2023-10-30T12:27:53.698 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: ok, hit_set contains 200:602f83fe:::foo:head 2023-10-30T12:27:53.698 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10116 ms) 2023-10-30T12:27:53.699 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-10-30T12:27:53.699 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg_num = 32 2023-10-30T12:27:53.699 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 0 ls 1698668869,0 2023-10-30T12:27:53.699 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 1 ls 1698668869,0 2023-10-30T12:27:53.699 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 2 ls 1698668869,0 2023-10-30T12:27:53.700 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 3 ls 1698668869,0 2023-10-30T12:27:53.700 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 4 ls 1698668869,0 2023-10-30T12:27:53.700 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 5 ls 1698668869,0 2023-10-30T12:27:53.700 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 6 ls 1698668869,0 2023-10-30T12:27:53.700 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 7 ls 1698668869,0 2023-10-30T12:27:53.700 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 8 ls 1698668869,0 2023-10-30T12:27:53.701 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 9 ls 1698668869,0 2023-10-30T12:27:53.701 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 10 ls 1698668869,0 2023-10-30T12:27:53.701 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 11 ls 1698668869,0 2023-10-30T12:27:53.701 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 12 ls 1698668869,0 2023-10-30T12:27:53.701 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 13 ls 1698668869,0 2023-10-30T12:27:53.702 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 14 ls 1698668869,0 2023-10-30T12:27:53.702 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 15 ls 1698668869,0 2023-10-30T12:27:53.702 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 16 ls 1698668869,0 2023-10-30T12:27:53.702 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 17 ls 1698668869,0 2023-10-30T12:27:53.702 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 18 ls 1698668869,0 2023-10-30T12:27:53.703 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 19 ls 1698668869,0 2023-10-30T12:27:53.703 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 20 ls 1698668869,0 2023-10-30T12:27:53.703 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 21 ls 1698668869,0 2023-10-30T12:27:53.703 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 22 ls 1698668869,0 2023-10-30T12:27:53.703 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 23 ls 1698668869,0 2023-10-30T12:27:53.703 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 24 ls 1698668869,0 2023-10-30T12:27:53.704 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 25 ls 1698668869,0 2023-10-30T12:27:53.704 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 26 ls 1698668869,0 2023-10-30T12:27:53.704 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 27 ls 1698668869,0 2023-10-30T12:27:53.704 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 28 ls 1698668869,0 2023-10-30T12:27:53.704 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 29 ls 1698668869,0 2023-10-30T12:27:53.704 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 30 ls 1698668869,0 2023-10-30T12:27:53.705 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg 31 ls 1698668869,0 2023-10-30T12:27:53.705 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: pg_num = 32 2023-10-30T12:27:53.705 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:6cac518f:::0:head 2023-10-30T12:27:53.705 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:02547ec2:::1:head 2023-10-30T12:27:53.705 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:f905c69b:::2:head 2023-10-30T12:27:53.706 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:cfc208b3:::3:head 2023-10-30T12:27:53.706 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d83876eb:::4:head 2023-10-30T12:27:53.706 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:b29083e3:::5:head 2023-10-30T12:27:53.706 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c4fdafeb:::6:head 2023-10-30T12:27:53.706 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:5c6b0b28:::7:head 2023-10-30T12:27:53.707 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:bd63b0f1:::8:head 2023-10-30T12:27:53.707 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:e960b815:::9:head 2023-10-30T12:27:53.707 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:52ea6a34:::10:head 2023-10-30T12:27:53.707 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:89d3ae78:::11:head 2023-10-30T12:27:53.707 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:de5d7c5f:::12:head 2023-10-30T12:27:53.708 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:566253c9:::13:head 2023-10-30T12:27:53.708 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:62a1935d:::14:head 2023-10-30T12:27:53.708 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:863748b0:::15:head 2023-10-30T12:27:53.708 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:3958e169:::16:head 2023-10-30T12:27:53.708 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:4d4dabf9:::17:head 2023-10-30T12:27:53.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:8391935d:::18:head 2023-10-30T12:27:53.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:28883081:::19:head 2023-10-30T12:27:53.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:69259c59:::20:head 2023-10-30T12:27:53.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:4bdb80b7:::21:head 2023-10-30T12:27:53.709 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:a11c5d71:::22:head 2023-10-30T12:27:53.710 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:271af37b:::23:head 2023-10-30T12:27:53.710 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:95b121be:::24:head 2023-10-30T12:27:53.710 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:58d1031b:::25:head 2023-10-30T12:27:53.710 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:0a050783:::26:head 2023-10-30T12:27:53.710 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c709704c:::27:head 2023-10-30T12:27:53.711 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:cbe56eaf:::28:head 2023-10-30T12:27:53.711 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:86b4b162:::29:head 2023-10-30T12:27:53.711 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:70d89383:::30:head 2023-10-30T12:27:53.711 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:dd450c7c:::31:head 2023-10-30T12:27:53.711 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:6d5729b1:::32:head 2023-10-30T12:27:53.712 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c388f3fb:::33:head 2023-10-30T12:27:53.712 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:56cfea31:::34:head 2023-10-30T12:27:53.712 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:9dbc1bf7:::35:head 2023-10-30T12:27:53.712 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:40b74ccd:::36:head 2023-10-30T12:27:53.712 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:4d5aaf42:::37:head 2023-10-30T12:27:53.712 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:920f362c:::38:head 2023-10-30T12:27:53.713 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:6cc53222:::39:head 2023-10-30T12:27:53.713 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:9cad833f:::40:head 2023-10-30T12:27:53.713 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:1ea84d41:::41:head 2023-10-30T12:27:53.713 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c4480ef6:::42:head 2023-10-30T12:27:53.713 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:a694361e:::43:head 2023-10-30T12:27:53.714 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d1bd33e9:::44:head 2023-10-30T12:27:53.714 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:ddc2cd5d:::45:head 2023-10-30T12:27:53.714 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:2b782207:::46:head 2023-10-30T12:27:53.714 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:7b187fca:::47:head 2023-10-30T12:27:53.714 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:90ecdf6f:::48:head 2023-10-30T12:27:53.715 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:a5ed95fe:::49:head 2023-10-30T12:27:53.715 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:ea0eaa55:::50:head 2023-10-30T12:27:53.715 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:f33ef17b:::51:head 2023-10-30T12:27:53.715 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:a0d1b2f6:::52:head 2023-10-30T12:27:53.715 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:60c5229e:::53:head 2023-10-30T12:27:53.715 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:edcbc575:::54:head 2023-10-30T12:27:53.716 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:102cf253:::55:head 2023-10-30T12:27:53.716 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:efb7fb0b:::56:head 2023-10-30T12:27:53.716 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:50d0a326:::57:head 2023-10-30T12:27:53.716 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d4dc5daf:::58:head 2023-10-30T12:27:53.716 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:3a130462:::59:head 2023-10-30T12:27:53.717 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:ec87ed71:::60:head 2023-10-30T12:27:53.717 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d5bc9454:::61:head 2023-10-30T12:27:53.717 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:3ddfe313:::62:head 2023-10-30T12:27:53.717 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:7c2816b9:::63:head 2023-10-30T12:27:53.717 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:47e00e4d:::64:head 2023-10-30T12:27:53.717 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c6410c18:::65:head 2023-10-30T12:27:53.718 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:b48ed237:::66:head 2023-10-30T12:27:53.718 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:cd63ad31:::67:head 2023-10-30T12:27:53.718 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:b179e92b:::68:head 2023-10-30T12:27:53.718 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:0d9f741a:::69:head 2023-10-30T12:27:53.718 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:6d3352ae:::70:head 2023-10-30T12:27:53.719 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c6d5c19e:::71:head 2023-10-30T12:27:53.719 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:bc4729c3:::72:head 2023-10-30T12:27:53.719 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:77e930b9:::73:head 2023-10-30T12:27:53.719 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:0abeecfd:::74:head 2023-10-30T12:27:53.719 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:b7c37e15:::75:head 2023-10-30T12:27:53.720 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:b6378398:::76:head 2023-10-30T12:27:53.720 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:02bd68de:::77:head 2023-10-30T12:27:53.720 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:cc795d2d:::78:head 2023-10-30T12:27:53.720 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:630d4fea:::79:head 2023-10-30T12:27:53.720 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:e0d29ef5:::80:head 2023-10-30T12:27:53.720 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:fd6f13d2:::81:head 2023-10-30T12:27:53.721 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:606461d5:::82:head 2023-10-30T12:27:53.721 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:eadbdc43:::83:head 2023-10-30T12:27:53.721 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:8761d0bb:::84:head 2023-10-30T12:27:53.721 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:9ef0186f:::85:head 2023-10-30T12:27:53.721 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:e0d41294:::86:head 2023-10-30T12:27:53.722 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:961de695:::87:head 2023-10-30T12:27:53.722 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:1423148f:::88:head 2023-10-30T12:27:53.722 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:633a8fa2:::89:head 2023-10-30T12:27:53.722 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:a8653809:::90:head 2023-10-30T12:27:53.722 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:3dac8b33:::91:head 2023-10-30T12:27:53.723 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:35aad435:::92:head 2023-10-30T12:27:53.723 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:f6dcc343:::93:head 2023-10-30T12:27:53.723 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:dbbdad87:::94:head 2023-10-30T12:27:53.723 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:1cb48ce0:::95:head 2023-10-30T12:27:53.723 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:03cd461c:::96:head 2023-10-30T12:27:53.723 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:17a4ea99:::97:head 2023-10-30T12:27:53.724 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:9993c9a7:::98:head 2023-10-30T12:27:53.724 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:6394211c:::99:head 2023-10-30T12:27:53.724 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:94c7ae57:::100:head 2023-10-30T12:27:53.724 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:6fdee5bb:::101:head 2023-10-30T12:27:53.724 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:9a477fd1:::102:head 2023-10-30T12:27:53.725 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:eb850916:::103:head 2023-10-30T12:27:53.725 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:affc56b9:::104:head 2023-10-30T12:27:53.725 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:b42dc814:::105:head 2023-10-30T12:27:53.725 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:f319f8f0:::106:head 2023-10-30T12:27:53.725 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:9a40b9de:::107:head 2023-10-30T12:27:53.725 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:8b524f28:::108:head 2023-10-30T12:27:53.726 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:e3de589f:::109:head 2023-10-30T12:27:53.726 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:90f90a5b:::110:head 2023-10-30T12:27:53.726 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:a7b4f1d7:::111:head 2023-10-30T12:27:53.726 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:af51766e:::112:head 2023-10-30T12:27:53.726 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:b6f90bd1:::113:head 2023-10-30T12:27:53.727 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:e0261208:::114:head 2023-10-30T12:27:53.727 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c9569ef7:::115:head 2023-10-30T12:27:53.727 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:61bebe50:::116:head 2023-10-30T12:27:53.727 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:fe93412b:::117:head 2023-10-30T12:27:53.727 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d3d38bee:::118:head 2023-10-30T12:27:53.727 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:3100ba0c:::119:head 2023-10-30T12:27:53.728 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d0560ada:::120:head 2023-10-30T12:27:53.728 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:f0ea8b35:::121:head 2023-10-30T12:27:53.728 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:766f231a:::122:head 2023-10-30T12:27:53.728 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:a07a2582:::123:head 2023-10-30T12:27:53.728 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:bd7c6b3a:::124:head 2023-10-30T12:27:53.728 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:fb2ddaff:::125:head 2023-10-30T12:27:53.729 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:4408e1fe:::126:head 2023-10-30T12:27:53.729 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:ee1df7a7:::127:head 2023-10-30T12:27:53.729 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c3002909:::128:head 2023-10-30T12:27:53.729 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:4f48ffa9:::129:head 2023-10-30T12:27:53.729 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:edf38733:::130:head 2023-10-30T12:27:53.730 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c08425c0:::131:head 2023-10-30T12:27:53.730 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:5f902d98:::132:head 2023-10-30T12:27:53.730 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:41ea2c93:::133:head 2023-10-30T12:27:53.730 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:813cee13:::134:head 2023-10-30T12:27:53.730 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:0131818d:::135:head 2023-10-30T12:27:53.730 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:26ba5a85:::136:head 2023-10-30T12:27:53.731 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:381b8a5a:::137:head 2023-10-30T12:27:53.731 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:28797e47:::138:head 2023-10-30T12:27:53.731 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:bfca7f22:::139:head 2023-10-30T12:27:53.731 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:36807075:::140:head 2023-10-30T12:27:53.731 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:80b03975:::141:head 2023-10-30T12:27:53.732 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:5c15709b:::142:head 2023-10-30T12:27:53.732 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:f39ea15e:::143:head 2023-10-30T12:27:53.732 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:ea992956:::144:head 2023-10-30T12:27:53.732 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:48887b1c:::145:head 2023-10-30T12:27:53.732 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:9f24a9dd:::146:head 2023-10-30T12:27:53.732 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:987f100b:::147:head 2023-10-30T12:27:53.733 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d2dd3581:::148:head 2023-10-30T12:27:53.733 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:7fed1808:::149:head 2023-10-30T12:27:53.733 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c80b70e9:::150:head 2023-10-30T12:27:53.733 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:85ed90f9:::151:head 2023-10-30T12:27:53.733 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:36428b24:::152:head 2023-10-30T12:27:53.734 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d044c34a:::153:head 2023-10-30T12:27:53.734 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:7c18bf58:::154:head 2023-10-30T12:27:53.734 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d1c21232:::155:head 2023-10-30T12:27:53.734 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:a7a3c575:::156:head 2023-10-30T12:27:53.734 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:87da0633:::157:head 2023-10-30T12:27:53.735 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d5ac3822:::158:head 2023-10-30T12:27:53.735 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:3f20522d:::159:head 2023-10-30T12:27:53.735 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:6ca26563:::160:head 2023-10-30T12:27:53.735 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:532ce135:::161:head 2023-10-30T12:27:53.735 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:c78863e6:::162:head 2023-10-30T12:27:58.920 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:13.993 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:58.971 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:14.045 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:58.973 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:14.020 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:58.975 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:14.022 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:58.999 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:14.046 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:58.999 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:14.072 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:59.022 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:14.096 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:59.025 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:14.098 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:59.050 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:14.123 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:00.113 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:15.187 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:00.115 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:15.188 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:00.128 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:15.201 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:00.167 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:15.241 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:00.169 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:15.216 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:00.169 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:15.242 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:00.172 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:15.219 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:00.182 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:15.256 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:00.186 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:15.233 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:18.945 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:34.018 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:18.946 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:34.020 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:18.970 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:34.044 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:18.992 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:34.066 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:18.994 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:34.041 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:18.996 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:34.043 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:18.998 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:34.072 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:19.020 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:34.067 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:19.023 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:34.097 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:39.135 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:54.209 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:39.144 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:54.218 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:39.183 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:11:54.256 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:39.194 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:54.241 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:39.195 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:54.242 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:39.199 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:54.273 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:39.205 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:54.279 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:39.224 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:11:54.271 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:39.238 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:11:54.311 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:59.130 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:14.203 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:59.132 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:14.206 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:59.156 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:14.230 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:59.179 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:14.253 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:59.180 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:14.227 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:59.182 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:14.229 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:59.182 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:14.256 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:59.204 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:14.251 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:59.207 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:14.280 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:03.670 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:7462ddf6:::163:head 2023-10-30T12:29:03.670 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:5d165639:::164:head 2023-10-30T12:29:03.670 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:f43765fc:::165:head 2023-10-30T12:29:03.670 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:b4c720e9:::166:head 2023-10-30T12:29:03.671 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:e694b040:::167:head 2023-10-30T12:29:03.671 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:afa38db2:::168:head 2023-10-30T12:29:03.671 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:77ba9f53:::169:head 2023-10-30T12:29:03.671 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:87495034:::170:head 2023-10-30T12:29:03.671 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:7c96bf0e:::171:head 2023-10-30T12:29:03.672 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:dbe346cc:::172:head 2023-10-30T12:29:03.672 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:e943ec24:::173:head 2023-10-30T12:29:03.672 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:f97a9c0c:::174:head 2023-10-30T12:29:03.672 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:6f26e74d:::175:head 2023-10-30T12:29:03.672 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:4f95e106:::176:head 2023-10-30T12:29:03.673 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:0e6f2f8f:::177:head 2023-10-30T12:29:03.673 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:05db05f1:::178:head 2023-10-30T12:29:03.673 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:38a78d66:::179:head 2023-10-30T12:29:03.673 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:d095610b:::180:head 2023-10-30T12:29:03.673 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:a1a9d709:::181:head 2023-10-30T12:29:03.674 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:1e5d39db:::182:head 2023-10-30T12:29:03.674 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:f7df4fb9:::183:head 2023-10-30T12:29:03.674 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:03a7f161:::184:head 2023-10-30T12:29:03.674 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:ba70721e:::185:head 2023-10-30T12:29:03.674 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:28e5662d:::186:head 2023-10-30T12:29:03.674 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:973d52de:::187:head 2023-10-30T12:29:03.675 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:4303eb1c:::188:head 2023-10-30T12:29:03.675 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:b990b48e:::189:head 2023-10-30T12:29:03.675 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:29b8165b:::190:head 2023-10-30T12:29:03.675 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:3547f197:::191:head 2023-10-30T12:29:03.675 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:7e260936:::192:head 2023-10-30T12:29:03.676 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:1abec7b1:::193:head 2023-10-30T12:29:03.676 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:10fdda93:::194:head 2023-10-30T12:29:03.676 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:15817eea:::195:head 2023-10-30T12:29:03.676 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:770bab57:::196:head 2023-10-30T12:29:03.676 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:ed9e13e7:::197:head 2023-10-30T12:29:03.677 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:71471a8f:::198:head 2023-10-30T12:29:03.677 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: checking for 201:10fb1d02:::199:head 2023-10-30T12:29:03.677 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21321 ms) 2023-10-30T12:29:03.677 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-10-30T12:29:03.677 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,0 2023-10-30T12:29:03.677 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: first is 1698668891 2023-10-30T12:29:03.678 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,0 2023-10-30T12:29:03.678 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,0 2023-10-30T12:29:03.678 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,1698668893,1698668894,0 2023-10-30T12:29:03.678 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,1698668893,1698668894,0 2023-10-30T12:29:03.678 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,1698668893,1698668894,0 2023-10-30T12:29:03.679 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,1698668893,1698668894,1698668896,1698668897,0 2023-10-30T12:29:03.679 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,1698668893,1698668894,1698668896,1698668897,0 2023-10-30T12:29:03.679 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,1698668893,1698668894,1698668896,1698668897,0 2023-10-30T12:29:03.679 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,1698668893,1698668894,1698668896,1698668897,1698668899,1698668900,0 2023-10-30T12:29:03.679 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,1698668893,1698668894,1698668896,1698668897,1698668899,1698668900,0 2023-10-30T12:29:03.679 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668891,1698668893,1698668894,1698668896,1698668897,1698668899,1698668900,0 2023-10-30T12:29:03.680 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698668894,1698668896,1698668897,1698668899,1698668900,1698668902,1698668903,0 2023-10-30T12:29:03.680 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: first now 1698668894, trimmed 2023-10-30T12:29:03.680 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22136 ms) 2023-10-30T12:29:03.680 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-10-30T12:29:03.680 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: foo0 2023-10-30T12:29:03.680 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-10-30T12:29:03.681 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (16101 ms) 2023-10-30T12:29:03.681 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-10-30T12:29:04.260 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:19.333 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:04.263 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:19.337 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:04.277 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:19.351 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:04.318 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:19.365 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:04.323 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:19.371 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:04.325 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:19.399 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:04.327 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:19.401 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:04.344 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:19.391 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:04.350 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:19.423 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.273 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:34.346 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.275 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:34.349 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.304 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:34.377 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.319 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:34.367 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.322 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:34.370 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.324 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:34.398 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.327 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:34.401 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.346 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:34.394 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.353 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:34.427 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.997 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:35.044 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:19.998 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:35.046 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:20.002 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:35.076 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:20.003 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:35.077 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:20.007 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:35.054 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:20.012 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:35.086 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:20.026 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:35.099 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:20.027 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:35.101 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:20.036 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:35.110 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:34.298 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:49.371 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:34.349 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:49.396 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:34.350 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:49.397 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:34.361 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:49.434 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:34.378 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:49.425 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:34.396 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:49.470 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:34.428 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:49.502 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:34.434 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:49.507 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:34.460 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:49.534 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:39.371 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:54.445 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:39.374 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:54.448 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:39.389 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:12:54.463 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:39.420 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:54.467 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:39.422 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:54.469 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:39.422 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:54.496 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:39.425 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:54.498 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:39.435 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:12:54.482 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:39.437 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:12:54.511 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:49.410 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:04.457 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:49.411 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:04.459 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:49.414 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:04.487 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:49.416 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:04.490 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:49.432 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:04.479 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:49.445 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:04.518 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:49.477 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:04.551 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:49.480 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:04.553 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:49.503 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:04.576 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:59.427 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:14.500 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:59.431 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:14.505 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:59.463 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:14.536 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:59.475 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:14.522 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:59.478 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:14.525 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:59.498 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:14.546 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:59.500 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:14.574 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:59.504 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:14.577 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:59.523 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:14.596 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:04.592 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:19.665 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:04.594 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:19.668 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:04.605 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:19.678 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:04.639 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:19.686 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:04.640 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:19.687 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:04.650 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:19.723 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:04.650 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:19.698 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:04.651 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:19.725 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:04.664 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:19.738 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:19.445 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:34.519 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:19.509 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:34.583 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:19.546 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:34.594 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:19.550 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:34.597 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:19.553 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:34.627 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:19.556 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:34.630 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:19.561 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:34.635 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:19.570 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:34.617 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:19.580 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:34.654 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:24.461 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:39.534 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:24.507 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:39.554 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:24.508 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:39.555 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:24.509 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:39.583 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:24.511 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:39.584 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:24.511 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:39.584 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:24.522 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:39.569 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:24.526 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:39.599 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:24.530 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:39.604 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:39.671 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:54.744 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:39.692 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:54.765 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:39.712 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:13:54.786 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:39.716 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:54.764 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:39.718 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:54.765 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:39.719 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:54.793 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:39.724 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:54.797 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:39.739 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:13:54.787 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:39.749 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:13:54.820 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:54.807 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:09.854 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:54.808 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:09.855 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:54.815 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:09.888 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:54.817 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:09.891 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:54.835 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:09.882 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:54.851 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:09.924 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:54.892 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:09.966 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:54.897 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:09.971 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:54.933 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:10.006 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:59.870 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:14.944 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:59.874 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:14.947 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:59.898 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:14.972 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:59.938 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:15.012 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:59.946 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:15.019 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:59.954 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:15.002 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:59.962 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:15.009 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:59.968 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:15.042 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:59.979 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:15.027 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:19.896 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:34.969 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:19.899 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:34.972 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:19.916 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:34.990 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:19.929 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:34.976 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:19.930 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:34.977 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:19.932 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:35.006 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:19.939 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:35.013 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:19.948 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:34.995 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:19.957 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:35.030 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:30.077 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:45.150 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:30.079 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:45.153 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:30.106 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:14:45.179 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:30.127 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:45.174 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:30.129 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:45.177 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:30.141 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:45.215 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:30.143 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:45.217 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:30.149 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:14:45.196 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:30.162 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:14:45.236 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:45.113 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:00.187 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:45.117 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:00.190 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:45.137 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:00.210 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:45.168 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:00.215 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:45.169 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:00.216 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:45.170 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:00.243 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:45.172 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:00.246 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:45.187 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:00.235 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:45.191 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:00.265 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:00.044 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:15.117 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:00.048 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:15.122 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:00.074 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:15.148 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:00.094 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:15.141 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:00.097 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:15.144 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:00.098 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:15.171 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:00.101 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:15.175 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:00.124 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:15.171 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:00.192 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:15.265 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:05.060 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:20.134 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:05.064 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:20.138 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:05.078 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:20.152 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:05.112 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:20.159 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:05.113 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:20.187 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:05.113 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:20.160 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:05.117 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:20.190 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:05.123 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:20.170 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:05.128 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:20.202 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:15.142 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:30.215 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:15.145 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:30.218 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:15.161 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:30.235 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:15.192 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:30.239 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:15.195 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:30.242 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:15.198 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:30.271 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:15.199 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:30.273 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:15.210 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:30.257 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:15.220 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:30.294 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:18.241 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:32:18.226+0000 23a89640 -1 snap_mapper.add_oid found existing snaps mapped on 219:2dc9a289:test-rados-api-smithi148-92193-80::foo:2, removing 2023-10-30T12:32:18.257 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T12:32:18.249+0000 28292640 -1 snap_mapper.add_oid found existing snaps mapped on 219:2dc9a289:test-rados-api-smithi148-92193-80::foo:2, removing 2023-10-30T12:32:25.180 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:40.253 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:25.183 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:40.257 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:25.210 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:40.283 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:25.227 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:40.274 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:25.229 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:40.276 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:25.240 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:40.314 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:25.245 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:40.319 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:25.246 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:40.293 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:25.273 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:40.347 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.169 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:45.243 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.225 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:45.299 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.242 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:45.316 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.269 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:45.316 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.273 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:45.321 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.286 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:45.360 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.290 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:45.337 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.307 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:45.380 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.326 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:45.400 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.988 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:46.035 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:30.988 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:46.036 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:31.000 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:15:46.048 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:31.200 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:46.274 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:31.201 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:46.275 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:31.213 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:15:46.287 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:31.277 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:46.351 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:31.278 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:46.352 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:31.290 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:15:46.364 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:45.284 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:00.357 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:45.289 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:00.363 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:45.327 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:00.400 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:45.350 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:00.397 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:45.352 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:00.400 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:45.361 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:00.435 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:45.365 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:00.438 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:45.389 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:00.436 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:45.409 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:00.482 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:46.057 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:01.104 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:46.058 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:01.105 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:46.066 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:01.113 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:46.171 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:01.245 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:46.181 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:01.254 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:46.197 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:01.270 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:46.322 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:01.396 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:46.326 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:01.399 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:46.349 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:01.422 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:56.176 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (22010 ms) 2023-10-30T12:32:56.177 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-10-30T12:32:56.177 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22924 ms) 2023-10-30T12:32:56.177 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-10-30T12:32:56.178 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4333 ms) 2023-10-30T12:32:56.178 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-10-30T12:32:56.178 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.178 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4718 ms) 2023-10-30T12:32:56.178 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-10-30T12:32:56.179 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4870 ms) 2023-10-30T12:32:56.179 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-10-30T12:32:56.179 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.179 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4516 ms) 2023-10-30T12:32:56.179 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-10-30T12:32:56.180 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.180 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5290 ms) 2023-10-30T12:32:56.180 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-10-30T12:32:56.180 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.180 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39213 ms) 2023-10-30T12:32:56.181 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-10-30T12:32:56.181 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.181 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18289 ms) 2023-10-30T12:32:56.181 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-10-30T12:32:56.181 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.181 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5278 ms) 2023-10-30T12:32:56.182 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-10-30T12:32:56.182 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.182 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4324 ms) 2023-10-30T12:32:56.182 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-10-30T12:32:56.182 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.183 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26743 ms) 2023-10-30T12:32:56.183 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-10-30T12:32:56.183 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.183 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15981 ms) 2023-10-30T12:32:56.183 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-10-30T12:32:56.184 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.184 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (7005 ms) 2023-10-30T12:32:56.184 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-10-30T12:32:56.184 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.184 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (6035 ms) 2023-10-30T12:32:56.185 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-10-30T12:32:56.185 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.185 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: waiting for scrubs... 2023-10-30T12:32:56.185 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: done waiting 2023-10-30T12:32:56.186 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26564 ms) 2023-10-30T12:32:56.186 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-10-30T12:32:56.186 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.186 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12388 ms) 2023-10-30T12:32:56.186 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-10-30T12:32:56.187 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.187 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (10733 ms) 2023-10-30T12:32:56.187 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-10-30T12:32:56.187 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:32:56.187 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11166 ms) 2023-10-30T12:32:56.188 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-10-30T12:33:00.464 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:15.537 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:00.516 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:15.589 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:00.547 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:15.617 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:00.558 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:15.605 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:00.560 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:15.608 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:00.575 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:15.648 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:00.576 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:15.649 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:00.588 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:15.635 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:00.609 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:15.682 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:05.501 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:20.574 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:05.503 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:20.577 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:05.517 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:20.591 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:05.566 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:20.613 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:05.572 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:20.646 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:05.573 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:20.620 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:05.574 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:20.648 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:05.590 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:20.637 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:05.591 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:20.664 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:15.393 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:30.466 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:15.443 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:30.516 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:15.451 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:30.499 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:15.453 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:30.501 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:15.465 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:30.539 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:15.475 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:30.522 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:15.491 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:30.564 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:15.493 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:30.566 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:15.514 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:30.588 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:25.541 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:40.614 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:25.543 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:40.617 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:25.562 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:40.636 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:25.586 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:40.633 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:25.587 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:40.634 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:25.590 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:40.664 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:25.593 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:40.667 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:25.605 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:40.652 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:25.614 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:40.688 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:30.493 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:45.566 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:30.495 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:45.568 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:16:45.577 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:30.540 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:45.587 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:30.541 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:45.614 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:30.542 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:45.590 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:30.543 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:45.617 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:30.553 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:16:45.600 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:30.554 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:16:45.628 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:50.705 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:05.779 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:50.708 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:05.782 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:50.734 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:05.808 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:50.754 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:05.801 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:50.755 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:05.802 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:50.767 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:05.841 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:50.769 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:05.843 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:50.776 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:05.824 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:50.812 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:05.885 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:00.688 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T12:34:00.680+0000 2ca9b640 -1 snap_mapper.add_oid found existing snaps mapped on 75:312956ac:test-rados-api-smithi148-92193-97::foo:31, removing 2023-10-30T12:34:00.703 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T12:34:00.663+0000 23a89640 -1 snap_mapper.add_oid found existing snaps mapped on 75:312956ac:test-rados-api-smithi148-92193-97::foo:31, removing 2023-10-30T12:34:05.616 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:20.689 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:05.617 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:20.691 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:05.635 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:20.708 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:05.664 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:20.711 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:05.665 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:20.739 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:05.666 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:20.713 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:05.667 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:20.741 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:05.684 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:20.731 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:05.686 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:20.760 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:10.723 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:25.796 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:10.724 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:25.771 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:10.727 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:25.774 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:10.730 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:25.804 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:10.742 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:25.789 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:10.746 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:25.820 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:10.803 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:25.876 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:10.804 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:25.878 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:10.824 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:25.897 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:20.659 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:35.732 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:20.663 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:35.737 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:20.685 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:35.759 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:20.711 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:35.758 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:20.712 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:35.759 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:20.712 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:35.786 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:20.716 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:35.790 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:20.731 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:35.778 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:20.738 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:35.812 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:30.792 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:45.866 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:30.795 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:45.869 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:30.821 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:45.895 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:30.857 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:45.904 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:30.861 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:45.908 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:30.862 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:45.935 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:30.863 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:45.936 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:30.887 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:45.960 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:30.887 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:45.934 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:35.828 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:50.901 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:35.884 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:50.957 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:35.923 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:50.997 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:35.945 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:50.992 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:35.948 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:50.995 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:35.948 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:51.021 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:35.952 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:51.025 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:35.965 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:51.039 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:35.966 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:51.013 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.247 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:53.321 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.249 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:53.323 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.265 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:17:53.338 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.297 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:53.344 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.298 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:53.345 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.312 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:17:53.359 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.321 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:53.394 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.323 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:53.396 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.338 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:17:53.411 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:50.733 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:05.807 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:50.787 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:05.834 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:50.789 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:05.862 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:50.790 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:05.837 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:50.815 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:05.862 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:50.819 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:05.893 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:50.839 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:05.913 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:50.842 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:05.916 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:50.869 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:05.942 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:51.604 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:06.651 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:51.605 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:06.652 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:51.613 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:06.660 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:51.696 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:06.770 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:51.698 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:06.771 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:51.710 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:06.783 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:51.745 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:06.819 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:51.746 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:06.820 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:51.755 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:06.829 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:05.843 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:20.917 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:05.847 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:20.921 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:05.893 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:20.940 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:05.896 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:20.943 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:05.897 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:20.970 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:05.912 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:20.985 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:05.916 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:20.990 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:05.925 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:20.972 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:05.948 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:21.021 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:10.811 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:25.884 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:10.858 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:25.931 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:10.924 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:25.961 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:10.925 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:25.952 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:10.925 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:25.953 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:10.925 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:25.966 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:10.925 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:25.988 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:10.926 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:25.989 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:10.930 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:26.004 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:20.893 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:35.966 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:20.897 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:35.970 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:20.921 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:35.995 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:20.943 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:35.990 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:20.945 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:35.993 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:20.946 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:36.020 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:20.950 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:36.024 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:20.967 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:36.014 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:20.973 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:36.047 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:31.028 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:46.101 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:31.030 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:46.103 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:31.053 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:46.126 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:31.075 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:46.122 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:31.078 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:46.152 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:31.079 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:46.126 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:31.079 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:46.153 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:31.102 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:46.149 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:31.104 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:46.177 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:40.949 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:56.022 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:40.952 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:56.025 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:40.971 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:18:56.044 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:40.998 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:56.045 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:40.999 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:56.047 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:41.002 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:56.076 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:41.003 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:56.077 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:41.018 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:18:56.066 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:41.023 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:18:56.097 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:50.979 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:06.053 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:50.982 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:06.055 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:50.998 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:06.072 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:51.027 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:06.101 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:51.030 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:06.077 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:51.030 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:06.104 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:51.031 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:06.079 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:51.045 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:06.118 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:51.047 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:06.094 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:55.993 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:11.066 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:55.997 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:11.071 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:56.010 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:11.083 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:56.045 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:11.118 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:56.046 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:11.093 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:56.047 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:11.121 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:56.048 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:11.095 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:56.058 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:11.105 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:56.058 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:11.132 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:11.182 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:26.255 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:11.186 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:26.259 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:11.213 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:26.287 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:11.233 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:26.280 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:11.237 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:26.284 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:11.249 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:26.322 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:11.250 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:26.324 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:11.268 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:26.315 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:11.277 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:26.351 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:26.184 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:41.257 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:26.186 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:41.260 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:26.215 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:41.288 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:26.232 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:41.279 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:26.235 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:41.283 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:26.249 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:41.323 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:26.250 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:41.324 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:26.263 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:41.310 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:26.288 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:41.362 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:31.179 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:46.253 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:31.235 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:46.309 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:31.253 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:46.327 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:31.284 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:46.358 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:31.287 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:46.361 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:31.289 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:46.336 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:31.293 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:46.340 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:31.302 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:46.376 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:31.315 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:46.362 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:37.996 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:53.070 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:38.001 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:53.075 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:38.014 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:53.088 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:38.017 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:53.064 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:38.060 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:53.134 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:38.064 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:53.111 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:38.087 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:53.161 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:38.090 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:53.164 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:38.097 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:53.144 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:41.166 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:56.239 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:41.170 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:56.244 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:41.185 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:19:56.258 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:41.217 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:56.264 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:41.220 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:56.267 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:41.220 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:56.294 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:41.225 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:56.298 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:41.231 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:19:56.278 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:41.237 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:19:56.311 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:51.150 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:06.223 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:51.195 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:06.242 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:51.196 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:06.270 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:51.197 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:06.244 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:51.221 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:06.268 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:51.222 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:06.295 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:51.258 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:06.332 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:51.262 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:06.336 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:51.285 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:06.359 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:51.781 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:36:51.781 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (10648 ms) 2023-10-30T12:36:51.781 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-10-30T12:36:51.782 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:36:51.782 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7193 ms) 2023-10-30T12:36:51.782 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-10-30T12:36:51.782 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:36:51.783 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6600 ms) 2023-10-30T12:36:51.783 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-10-30T12:36:51.783 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:36:51.783 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (27068 ms) 2023-10-30T12:36:51.783 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-10-30T12:36:51.784 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:36:51.784 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (16198 ms) 2023-10-30T12:36:51.784 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-10-30T12:36:51.784 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12894 ms) 2023-10-30T12:36:51.784 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-10-30T12:36:51.784 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-10-30T12:36:51.785 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: 00000010 69 73 |is| 2023-10-30T12:36:51.785 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: 00000012 2023-10-30T12:36:51.785 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (13155 ms) 2023-10-30T12:36:51.785 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-10-30T12:36:51.785 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:36:51.786 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6993 ms) 2023-10-30T12:36:51.786 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (788733 ms total) 2023-10-30T12:36:51.786 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: 2023-10-30T12:36:51.786 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-10-30T12:36:51.786 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-10-30T12:36:51.787 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1507 ms) 2023-10-30T12:36:51.787 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-10-30T12:36:51.787 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10925 ms) 2023-10-30T12:36:51.787 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-10-30T12:36:51.787 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19959 ms) 2023-10-30T12:36:51.788 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-10-30T12:36:51.788 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (18 ms) 2023-10-30T12:36:51.788 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (32409 ms total) 2023-10-30T12:36:51.788 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: 2023-10-30T12:36:51.788 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-10-30T12:36:51.789 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-10-30T12:36:51.789 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7867 ms) 2023-10-30T12:36:51.789 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-10-30T12:36:51.789 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (10028 ms) 2023-10-30T12:36:51.789 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-10-30T12:36:51.789 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: waiting for scrub... 2023-10-30T12:36:51.790 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: done waiting 2023-10-30T12:36:51.790 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (27336 ms) 2023-10-30T12:36:51.790 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-10-30T12:36:51.790 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10435 ms) 2023-10-30T12:36:51.790 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-10-30T12:36:51.791 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8851 ms) 2023-10-30T12:36:51.791 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-10-30T12:36:51.791 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8827 ms) 2023-10-30T12:36:51.791 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-10-30T12:36:51.791 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12053 ms) 2023-10-30T12:36:51.792 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-10-30T12:36:56.331 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:11.404 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:56.333 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:11.406 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:56.348 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:11.422 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:56.379 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:11.452 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:56.380 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:11.454 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:56.385 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:11.432 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:56.388 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:11.436 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:56.397 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:11.471 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:56.409 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:11.456 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:11.335 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:26.408 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:11.347 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:26.421 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:11.393 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:26.440 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:11.396 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:26.470 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:11.397 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:26.444 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:11.399 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:26.473 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:11.463 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:26.537 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:11.468 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:26.542 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:11.506 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:26.553 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:12.478 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:27.525 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:12.479 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:27.526 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:12.488 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:27.535 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:12.623 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:27.696 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:12.624 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:27.697 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:12.668 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:27.741 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:12.869 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:27.942 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:12.873 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:27.945 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:12.882 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:27.956 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:26.544 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:41.617 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:26.545 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:41.619 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:26.580 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:41.654 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:26.600 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:41.647 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:26.602 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:41.649 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:26.604 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:41.677 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:26.606 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:41.680 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:26.639 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:41.713 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:26.640 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:41.687 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:31.437 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:46.510 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:31.441 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:46.515 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:31.458 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:46.532 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:31.485 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:46.559 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:31.486 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:46.533 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:31.487 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:46.534 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:31.489 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:46.563 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:31.502 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:46.549 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:31.504 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:46.578 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:41.478 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:56.552 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:41.481 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:56.555 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:41.502 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:20:56.576 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:41.525 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:56.599 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:41.529 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:56.575 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:41.529 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:56.602 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:41.530 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:56.576 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:41.553 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:20:56.600 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:41.553 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:20:56.626 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:51.682 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:06.756 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:51.688 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:06.762 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:51.716 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:06.789 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:51.739 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:06.812 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:51.739 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:06.787 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:51.740 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:06.814 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:51.740 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:06.788 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:51.764 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:06.811 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:51.764 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:06.838 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:56.667 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:11.741 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:56.670 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:11.744 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:56.683 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:11.756 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:56.714 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:11.762 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:56.716 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:11.790 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:56.718 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:11.765 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:56.719 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:11.793 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:56.732 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:11.805 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:56.733 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:11.781 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:11.664 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:26.737 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:11.666 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:26.740 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:11.694 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:26.768 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:11.714 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:26.761 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:11.716 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:26.763 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:11.717 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:26.791 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:11.719 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:26.793 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:11.742 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:26.789 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:11.744 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:26.818 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:12.462 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:27.509 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:12.463 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:27.510 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:12.471 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:27.518 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:12.487 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:27.560 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:12.487 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:27.561 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:12.488 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:27.561 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:12.488 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:27.562 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:12.496 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:27.569 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:12.496 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:27.570 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:26.718 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:41.791 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:26.719 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:41.793 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:26.746 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:41.819 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:26.762 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:41.809 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:26.765 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:41.812 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:26.766 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:41.840 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:26.769 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:41.843 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:26.788 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:41.836 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:26.793 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:41.866 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:31.773 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:46.847 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:31.819 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:46.893 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:31.834 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:46.907 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:31.866 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:46.939 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:31.867 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:46.940 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:31.871 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:46.918 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:31.875 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:46.922 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:31.885 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:46.959 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:31.890 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:46.937 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:35.821 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:50.894 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:35.827 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:50.901 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:35.840 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:50.914 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:35.845 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:50.892 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:35.848 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:50.921 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:35.850 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:50.897 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:35.907 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:50.980 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:35.918 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:50.991 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:35.941 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:50.988 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:41.742 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:56.815 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:41.786 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:56.833 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:41.787 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:56.834 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:41.789 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:56.863 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:41.807 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:21:56.854 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:41.809 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:21:56.883 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:41.835 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:56.909 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:41.841 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:56.915 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:41.857 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:21:56.931 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:51.810 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:06.883 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:51.812 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:06.886 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:51.827 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:06.900 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:51.861 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:06.908 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:51.864 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:06.911 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:51.876 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:06.923 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:51.878 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:06.952 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:51.881 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:06.955 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:51.895 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:06.968 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:56.969 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:12.042 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:56.971 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:12.018 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:56.976 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:12.050 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:56.984 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:12.031 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:56.992 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:12.066 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:57.005 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:12.053 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:57.037 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:12.111 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:57.046 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:12.119 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:57.060 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:12.134 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:12.038 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:27.112 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:12.041 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:27.115 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:12.066 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:27.140 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:12.087 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:27.161 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:12.088 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:27.134 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:12.089 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:27.136 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:12.090 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:27.164 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:12.117 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:27.164 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:12.129 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:27.203 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:27.181 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:42.254 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:27.239 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:42.313 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:27.280 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:42.353 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:27.306 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:42.353 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:27.309 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:42.356 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:27.322 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:42.396 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:27.325 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:42.399 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:27.348 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:42.395 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:27.363 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:42.436 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:28.235 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:43.282 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:28.236 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:43.283 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:28.245 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:43.292 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:28.300 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:43.373 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:28.301 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:43.375 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:28.310 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:43.384 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:28.341 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:43.414 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:28.342 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:43.415 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:28.351 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:43.424 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:37.253 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:52.326 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:37.256 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:52.330 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:37.279 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:22:52.352 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:37.302 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:52.349 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:37.304 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:52.351 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:37.306 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:52.380 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:37.310 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:52.383 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:37.330 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:22:52.377 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:37.335 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:22:52.409 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:43.446 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (9443 ms) 2023-10-30T12:39:43.446 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-10-30T12:39:43.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13887 ms) 2023-10-30T12:39:43.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-10-30T12:39:43.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9950 ms) 2023-10-30T12:39:43.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-10-30T12:39:43.447 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (14097 ms) 2023-10-30T12:39:43.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-10-30T12:39:43.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9857 ms) 2023-10-30T12:39:43.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-10-30T12:39:43.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8695 ms) 2023-10-30T12:39:43.448 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-10-30T12:39:43.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: hmm, no HitSet yet 2023-10-30T12:39:43.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2023-10-30T12:39:43.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (10066 ms) 2023-10-30T12:39:43.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-10-30T12:39:43.449 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,0 2023-10-30T12:39:43.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: first is 1698669486 2023-10-30T12:39:43.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,0 2023-10-30T12:39:43.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,0 2023-10-30T12:39:43.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,1698669488,1698669489,0 2023-10-30T12:39:43.450 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,1698669488,1698669489,0 2023-10-30T12:39:43.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,1698669488,1698669489,0 2023-10-30T12:39:43.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,1698669488,1698669489,1698669492,1698669493,0 2023-10-30T12:39:43.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,1698669488,1698669489,1698669492,1698669493,0 2023-10-30T12:39:43.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,1698669488,1698669489,1698669492,1698669493,0 2023-10-30T12:39:43.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,1698669488,1698669489,1698669492,1698669493,1698669495,1698669496,0 2023-10-30T12:39:43.451 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,1698669488,1698669489,1698669492,1698669493,1698669495,1698669496,0 2023-10-30T12:39:43.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669486,1698669488,1698669489,1698669492,1698669493,1698669495,1698669496,0 2023-10-30T12:39:43.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: got ls 1698669489,1698669492,1698669493,1698669495,1698669496,1698669498,1698669499,0 2023-10-30T12:39:43.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: first now 1698669489, trimmed 2023-10-30T12:39:43.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21736 ms) 2023-10-30T12:39:43.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-10-30T12:39:43.452 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: foo0 2023-10-30T12:39:43.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-10-30T12:39:43.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (15866 ms) 2023-10-30T12:39:43.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-10-30T12:39:43.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20615 ms) 2023-10-30T12:39:43.453 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-10-30T12:39:43.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (24778 ms) 2023-10-30T12:39:43.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-10-30T12:39:43.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5158 ms) 2023-10-30T12:39:43.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-10-30T12:39:43.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4633 ms) 2023-10-30T12:39:43.454 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-10-30T12:39:43.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4410 ms) 2023-10-30T12:39:43.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-10-30T12:39:43.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4535 ms) 2023-10-30T12:39:43.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (263123 ms total) 2023-10-30T12:39:43.455 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: 2023-10-30T12:39:43.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-10-30T12:39:43.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1227801 ms total) 2023-10-30T12:39:43.456 INFO:tasks.workunit.client.0.smithi148.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-10-30T12:39:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91893 2023-10-30T12:39:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 91893 2023-10-30T12:39:43.460 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.460 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92094 2023-10-30T12:39:43.460 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92094 2023-10-30T12:39:43.460 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.460 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92632 2023-10-30T12:39:43.460 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92632 2023-10-30T12:39:43.461 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.461 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92331 2023-10-30T12:39:43.461 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92331 2023-10-30T12:39:43.461 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.461 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92192 2023-10-30T12:39:43.462 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92192 2023-10-30T12:39:43.462 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.462 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91977 2023-10-30T12:39:43.462 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 91977 2023-10-30T12:39:43.462 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.463 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92249 2023-10-30T12:39:43.463 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92249 2023-10-30T12:39:43.463 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.463 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92030 2023-10-30T12:39:43.463 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92030 2023-10-30T12:39:43.463 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.464 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92478 2023-10-30T12:39:43.464 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92478 2023-10-30T12:39:43.464 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.464 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91866 2023-10-30T12:39:43.464 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 91866 2023-10-30T12:39:43.465 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.465 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91930 2023-10-30T12:39:43.465 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 91930 2023-10-30T12:39:43.465 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.465 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92517 2023-10-30T12:39:43.466 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92517 2023-10-30T12:39:43.466 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.466 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91857 2023-10-30T12:39:43.466 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 91857 2023-10-30T12:39:43.467 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.467 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91878 2023-10-30T12:39:43.467 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 91878 2023-10-30T12:39:43.467 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.467 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92136 2023-10-30T12:39:43.468 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92136 2023-10-30T12:39:43.468 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.468 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92225 2023-10-30T12:39:43.468 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92225 2023-10-30T12:39:43.468 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.468 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92177 2023-10-30T12:39:43.469 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92177 2023-10-30T12:39:43.469 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.469 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92592 2023-10-30T12:39:43.469 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92592 2023-10-30T12:39:43.469 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.470 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92455 2023-10-30T12:39:43.470 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92455 2023-10-30T12:39:43.470 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.470 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92071 2023-10-30T12:39:43.470 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92071 2023-10-30T12:39:43.471 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.471 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92371 2023-10-30T12:39:43.471 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92371 2023-10-30T12:39:43.471 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.471 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91850 2023-10-30T12:39:43.472 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 91850 2023-10-30T12:39:43.472 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.472 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91910 2023-10-30T12:39:43.472 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 91910 2023-10-30T12:39:43.472 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.472 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92315 2023-10-30T12:39:43.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92315 2023-10-30T12:39:43.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=91957 2023-10-30T12:39:43.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 91957 2023-10-30T12:39:43.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.473 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92407 2023-10-30T12:39:43.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92407 2023-10-30T12:39:43.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:39:43.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ pid=92427 2023-10-30T12:39:43.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ wait 92427 2023-10-30T12:39:43.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ exit 0 2023-10-30T12:39:43.475 INFO:tasks.workunit.client.0.smithi148.stderr:+ cleanup 2023-10-30T12:39:43.475 INFO:tasks.workunit.client.0.smithi148.stderr:+ pkill -P 91845 2023-10-30T12:39:43.476 INFO:tasks.workunit.client.0.smithi148.stderr:+ true 2023-10-30T12:39:43.476 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:39:43.477 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:39:43.506 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-10-30T12:39:43.506 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T12:39:43.810 DEBUG:teuthology.parallel:result is None 2023-10-30T12:39:43.810 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T12:39:43.837 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T12:39:43.837 DEBUG:teuthology.orchestra.run.smithi148:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-10-30T12:39:43.900 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T12:39:43.901 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T12:39:43.911 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T12:39:43.912 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:39:44.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.039+0000 7ff24454e640 10 monclient: get_monmap_and_config 2023-10-30T12:39:44.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.039+0000 7ff24454e640 10 monclient: build_initial_monmap 2023-10-30T12:39:44.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.039+0000 7ff24454e640 10 monclient: monmap: 2023-10-30T12:39:44.041 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:44.041 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:44.041 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:44.040353+0000 2023-10-30T12:39:44.042 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:44.040353+0000 2023-10-30T12:39:44.042 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:44.042 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:44.042 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:44.043 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:44.043 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:44.043 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:44.043 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:44.043 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.039+0000 7ff24454e640 1 Processor -- start 2023-10-30T12:39:44.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.039+0000 7ff24454e640 1 -- start start 2023-10-30T12:39:44.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.039+0000 7ff24454e640 10 monclient: init 2023-10-30T12:39:44.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:44.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-10-30T12:39:44.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c0b7c10 0x7ff23c0a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 10 monclient(hunting): picked mon.noname-b con 0x7ff23c0b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:44.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff23c141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:44.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff23c13e730 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:44.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23c0a9350 con 0x7ff23c13e730 2023-10-30T12:39:44.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23c0aa6a0 con 0x7ff23c141940 2023-10-30T12:39:44.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:44.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c0670a0 con 0x7ff23c0b7c10 2023-10-30T12:39:44.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:44.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.040+0000 7ff24454e640 10 monclient(hunting): authenticate will time out at 2876.245549s 2023-10-30T12:39:44.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.041+0000 7ff2422ea640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c0b7c10 0x7ff23c0a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.041+0000 7ff242aeb640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff23c13e730 0x7ff23c13eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42426/0 (socket says 172.21.15.148:42426) 2023-10-30T12:39:44.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.041+0000 7ff242aeb640 1 -- 172.21.15.148:0/4279997465 learned_addr learned my addr 172.21.15.148:0/4279997465 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:39:44.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.041+0000 7ff2422ea640 10 monclient(hunting): get_auth_request con 0x7ff23c0b7c10 auth_method 0 2023-10-30T12:39:44.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.041+0000 7ff2422ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:44.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.041+0000 7ff2422ea640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:44.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.041+0000 7ff2422ea640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:44.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.042+0000 7ff2422ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:44.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.042+0000 7ff2422ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:44.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.042+0000 7ff2422ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:44.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.043+0000 7ff2422ea640 10 monclient(hunting): handle_auth_done global_id 17678 payload 274 2023-10-30T12:39:44.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.043+0000 7ff2422ea640 1 -- 172.21.15.148:0/4279997465 >> v1:172.21.15.150:6789/0 conn(0x7ff23c141940 legacy=0x7ff23c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:44.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.043+0000 7ff2422ea640 1 -- 172.21.15.148:0/4279997465 >> v1:172.21.15.148:6789/0 conn(0x7ff23c13e730 legacy=0x7ff23c13eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:44.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.043+0000 7ff2422ea640 10 monclient: _finish_hunting 0 2023-10-30T12:39:44.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.043+0000 7ff2422ea640 1 monclient: found mon.noname-b 2023-10-30T12:39:44.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.043+0000 7ff2422ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:44.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.043+0000 7ff2422ea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:44.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.043+0000 7ff2422ea640 1 -- 172.21.15.148:0/4279997465 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c0aa910 con 0x7ff23c0b7c10 2023-10-30T12:39:44.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.043+0000 7ff2412e8640 1 -- 172.21.15.148:0/4279997465 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060454627 0 0) 0x7ff238001110 con 0x7ff23c13e730 2023-10-30T12:39:44.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.043+0000 7ff2412e8640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:39:44.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.045+0000 7ff2422ea640 1 --2- 172.21.15.148:0/4279997465 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c0b7c10 0x7ff23c0a8aa0 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7ff234009dc0 tx=0x7ff23402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d1f89cb0ea3640a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:44.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 1 -- 172.21.15.148:0/4279997465 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23402cc40 con 0x7ff23c0b7c10 2023-10-30T12:39:44.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:44.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:44.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 10 monclient: dump: 2023-10-30T12:39:44.059 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:44.059 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:44.059 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.060 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.060 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:44.060 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:44.060 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:44.060 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:44.060 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:44.061 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 10 monclient: _finish_auth 0 2023-10-30T12:39:44.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 10 monclient: _check_auth_tickets 2023-10-30T12:39:44.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:44.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 1 -- 172.21.15.148:0/4279997465 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23402cdb0 con 0x7ff23c0b7c10 2023-10-30T12:39:44.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:44.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 1 -- 172.21.15.148:0/4279997465 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234034660 con 0x7ff23c0b7c10 2023-10-30T12:39:44.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 5 monclient: authenticate success, global_id 17678 2023-10-30T12:39:44.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:44.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 4 set_mon_vals no callback set 2023-10-30T12:39:44.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:44.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:44.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff2412e8640 10 monclient: dump: 2023-10-30T12:39:44.063 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:44.063 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:44.064 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.064 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.064 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:44.064 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:44.064 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:44.064 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:44.065 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:44.065 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 10 monclient: shutdown 2023-10-30T12:39:44.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:44.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 1 -- 172.21.15.148:0/4279997465 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c0b7c10 msgr2=0x7ff23c0a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:44.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 1 --2- 172.21.15.148:0/4279997465 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c0b7c10 0x7ff23c0a8aa0 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7ff234009dc0 tx=0x7ff23402be90 comp rx=0 tx=0).stop 2023-10-30T12:39:44.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 1 -- 172.21.15.148:0/4279997465 shutdown_connections 2023-10-30T12:39:44.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 1 --2- 172.21.15.148:0/4279997465 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c0b7c10 0x7ff23c0a8aa0 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 1 -- 172.21.15.148:0/4279997465 >> 172.21.15.148:0/4279997465 conn(0x7ff23c06c660 msgr2=0x7ff23c0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:44.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 1 -- 172.21.15.148:0/4279997465 shutdown_connections 2023-10-30T12:39:44.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.049+0000 7ff24454e640 1 -- 172.21.15.148:0/4279997465 wait complete. 2023-10-30T12:39:44.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.050+0000 7ff24454e640 10 monclient: build_initial_monmap 2023-10-30T12:39:44.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.050+0000 7ff24454e640 10 monclient: monmap: 2023-10-30T12:39:44.067 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:44.067 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:44.067 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:44.067 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:44.068 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:44.068 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:44.068 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:44.068 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:44.068 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:44.069 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.050+0000 7ff24454e640 1 librados: starting msgr at 2023-10-30T12:39:44.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.050+0000 7ff24454e640 1 librados: starting objecter 2023-10-30T12:39:44.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.050+0000 7ff24454e640 1 Processor -- start 2023-10-30T12:39:44.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 1 -- start start 2023-10-30T12:39:44.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 1 librados: setting wanted keys 2023-10-30T12:39:44.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 1 librados: calling monclient init 2023-10-30T12:39:44.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient: init 2023-10-30T12:39:44.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:44.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:39:44.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23c0b7c10 0x7ff23c1c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient(hunting): picked mon.noname-b con 0x7ff23c0b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:44.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c13e730 0x7ff23c1d6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient(hunting): picked mon.noname-c con 0x7ff23c13e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:44.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff23c141940 0x7ff23c1d7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient(hunting): picked mon.noname-a con 0x7ff23c141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:44.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:44.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c13c250 con 0x7ff23c141940 2023-10-30T12:39:44.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:44.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c1d7860 con 0x7ff23c0b7c10 2023-10-30T12:39:44.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:44.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c0b0f50 con 0x7ff23c13e730 2023-10-30T12:39:44.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:44.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff24454e640 10 monclient(hunting): authenticate will time out at 2876.256478s 2023-10-30T12:39:44.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.051+0000 7ff241ae9640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c13e730 0x7ff23c1d6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff242aeb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff23c141940 0x7ff23c1d7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff241ae9640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c13e730 0x7ff23c1d6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:53888/0 (socket says 172.21.15.148:53888) 2023-10-30T12:39:44.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff241ae9640 1 -- 172.21.15.148:0/3248775888 learned_addr learned my addr 172.21.15.148:0/3248775888 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:44.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff241ae9640 10 monclient(hunting): get_auth_request con 0x7ff23c13e730 auth_method 0 2023-10-30T12:39:44.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff241ae9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:44.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff241ae9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:44.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff241ae9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:44.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff2422ea640 1 --2- 172.21.15.148:0/3248775888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23c0b7c10 0x7ff23c1c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff242aeb640 10 monclient(hunting): get_auth_request con 0x7ff23c141940 auth_method 0 2023-10-30T12:39:44.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff242aeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:44.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff242aeb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:44.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff242aeb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:44.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff2422ea640 10 monclient(hunting): get_auth_request con 0x7ff23c0b7c10 auth_method 0 2023-10-30T12:39:44.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff2422ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:44.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff2422ea640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:44.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff2422ea640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:44.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff241ae9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:44.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff241ae9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:44.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff241ae9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:44.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff242aeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:44.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff242aeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:44.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.052+0000 7ff242aeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:44.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.053+0000 7ff2422ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:44.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.053+0000 7ff2422ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:44.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.053+0000 7ff2422ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:44.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.055+0000 7ff241ae9640 10 monclient(hunting): handle_auth_done global_id 17681 payload 824 2023-10-30T12:39:44.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.055+0000 7ff241ae9640 1 -- 172.21.15.148:0/3248775888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23c0b7c10 msgr2=0x7ff23c1c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:44.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.055+0000 7ff241ae9640 1 --2- 172.21.15.148:0/3248775888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23c0b7c10 0x7ff23c1c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.055+0000 7ff241ae9640 1 -- 172.21.15.148:0/3248775888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff23c141940 msgr2=0x7ff23c1d7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:44.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.055+0000 7ff241ae9640 1 --2- 172.21.15.148:0/3248775888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff23c141940 0x7ff23c1d7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.055+0000 7ff241ae9640 10 monclient: _finish_hunting 0 2023-10-30T12:39:44.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.055+0000 7ff241ae9640 1 monclient: found mon.noname-c 2023-10-30T12:39:44.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.055+0000 7ff241ae9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:44.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.055+0000 7ff241ae9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:39:44.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.055+0000 7ff241ae9640 1 -- 172.21.15.148:0/3248775888 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c1c85b0 con 0x7ff23c13e730 2023-10-30T12:39:44.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.057+0000 7ff241ae9640 1 --2- 172.21.15.148:0/3248775888 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c13e730 0x7ff23c1d6de0 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00c400 tx=0x7ff22c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:44.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.057+0000 7ff232ffd640 10 client.?.objecter ms_handle_connect 0x7ff23c13e730 2023-10-30T12:39:44.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.057+0000 7ff232ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:44.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 1 -- 172.21.15.148:0/3248775888 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0052d0 con 0x7ff23c13e730 2023-10-30T12:39:44.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:44.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:44.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 10 monclient: dump: 2023-10-30T12:39:44.082 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:44.082 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:44.083 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.083 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.083 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:44.083 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:44.084 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:44.084 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:44.084 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:44.084 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 10 monclient: _finish_auth 0 2023-10-30T12:39:44.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:39:44.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:44.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 1 -- 172.21.15.148:0/3248775888 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c00e040 con 0x7ff23c13e730 2023-10-30T12:39:44.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff24454e640 5 monclient: authenticate success, global_id 17681 2023-10-30T12:39:44.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff24454e640 10 monclient: _renew_subs 2023-10-30T12:39:44.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff24454e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:44.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff24454e640 1 -- 172.21.15.148:0/3248775888 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23c0aff30 con 0x7ff23c13e730 2023-10-30T12:39:44.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:44.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 1 -- 172.21.15.148:0/3248775888 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0035f0 con 0x7ff23c13e730 2023-10-30T12:39:44.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:44.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:44.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff232ffd640 10 monclient: dump: 2023-10-30T12:39:44.087 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:44.087 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:44.087 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.088 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.088 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:44.088 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:44.088 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:44.088 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:44.089 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:44.089 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff240ae7640 4 set_mon_vals no callback set 2023-10-30T12:39:44.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff24454e640 10 client.17681.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:44.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff24454e640 10 monclient: _renew_subs 2023-10-30T12:39:44.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff24454e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:44.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff24454e640 1 -- 172.21.15.148:0/3248775888 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23c1ce580 con 0x7ff23c13e730 2023-10-30T12:39:44.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.061+0000 7ff24454e640 1 librados: init done 2023-10-30T12:39:44.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.062+0000 7ff24454e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:44.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.062+0000 7ff24454e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:44.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.062+0000 7ff24454e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:44.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.062+0000 7ff24454e640 1 -- 172.21.15.148:0/3248775888 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23c0a97c0 con 0x7ff23c13e730 2023-10-30T12:39:44.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.081+0000 7ff232ffd640 1 -- 172.21.15.148:0/3248775888 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7ff22c003760 con 0x7ff23c13e730 2023-10-30T12:39:44.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.081+0000 7ff232ffd640 1 --2- 172.21.15.148:0/3248775888 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff22006c840 0x7ff22006ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.081+0000 7ff2422ea640 1 --2- 172.21.15.148:0/3248775888 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff22006c840 0x7ff22006ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.081+0000 7ff2422ea640 10 monclient: get_auth_request con 0x7ff22006c840 auth_method 0 2023-10-30T12:39:44.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.082+0000 7ff2422ea640 1 --2- 172.21.15.148:0/3248775888 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff22006c840 0x7ff22006ed00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff234036000 tx=0x7ff234030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:44.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.082+0000 7ff232ffd640 10 client.17681.objecter ms_handle_connect 0x7ff22006c840 2023-10-30T12:39:44.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.083+0000 7ff232ffd640 1 -- 172.21.15.148:0/3248775888 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(899..899 src has 501..899) v4 ==== 5133+0+0 (secure 0 0 0) 0x7ff22c057380 con 0x7ff23c13e730 2023-10-30T12:39:44.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.083+0000 7ff232ffd640 10 client.17681.objecter ms_dispatch 0x7ff23c000b90 osd_map(899..899 src has 501..899) v4 2023-10-30T12:39:44.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.083+0000 7ff232ffd640 3 client.17681.objecter handle_osd_map got epochs [899,899] > 0 2023-10-30T12:39:44.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.083+0000 7ff232ffd640 3 client.17681.objecter handle_osd_map decoding full epoch 899 2023-10-30T12:39:44.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.083+0000 7ff232ffd640 20 client.17681.objecter dump_active .. 0 homeless 2023-10-30T12:39:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.316+0000 7ff232ffd640 1 -- 172.21.15.148:0/3248775888 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff22c05b420 con 0x7ff23c13e730 2023-10-30T12:39:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.316+0000 7ff232ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.316+0000 7ff232ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:44.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.497+0000 7ff24454e640 1 -- 172.21.15.148:0/3248775888 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff23c13f290 con 0x7ff22006c840 2023-10-30T12:39:44.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.502+0000 7ff232ffd640 1 -- 172.21.15.148:0/3248775888 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151975 (secure 0 0 0) 0x7ff23404a4a0 con 0x7ff22006c840 2023-10-30T12:39:44.505 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:39:44.506 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T12:39:44.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 10 librados: watch_flush enter 2023-10-30T12:39:44.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 10 librados: watch_flush exit 2023-10-30T12:39:44.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 20 client.17681.objecter shutdown clearing up homeless session... 2023-10-30T12:39:44.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 10 client.17681.objecter successfully canceled tick 2023-10-30T12:39:44.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 -- 172.21.15.148:0/3248775888 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff22006c840 msgr2=0x7ff22006ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:44.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 --2- 172.21.15.148:0/3248775888 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff22006c840 0x7ff22006ed00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff234036000 tx=0x7ff234030000 comp rx=0 tx=0).stop 2023-10-30T12:39:44.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 10 monclient: shutdown 2023-10-30T12:39:44.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:44.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 -- 172.21.15.148:0/3248775888 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c13e730 msgr2=0x7ff23c1d6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:44.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 --2- 172.21.15.148:0/3248775888 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c13e730 0x7ff23c1d6de0 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00c400 tx=0x7ff22c00c8c0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 -- 172.21.15.148:0/3248775888 shutdown_connections 2023-10-30T12:39:44.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 --2- 172.21.15.148:0/3248775888 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff22006c840 0x7ff22006ed00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 --2- 172.21.15.148:0/3248775888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff23c141940 0x7ff23c1d7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 --2- 172.21.15.148:0/3248775888 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff23c13e730 0x7ff23c1d6de0 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 --2- 172.21.15.148:0/3248775888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23c0b7c10 0x7ff23c1c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 -- 172.21.15.148:0/3248775888 >> 172.21.15.148:0/3248775888 conn(0x7ff23c06c660 msgr2=0x7ff23c0b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:44.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 -- 172.21.15.148:0/3248775888 shutdown_connections 2023-10-30T12:39:44.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 -- 172.21.15.148:0/3248775888 wait complete. 2023-10-30T12:39:44.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.504+0000 7ff24454e640 1 librados: shutdown 2023-10-30T12:39:44.517 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1560,"stamp":"2023-10-30T12:39:43.390404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":552420,"kb_used_data":77820,"kb_used_omap":12,"kb_used_meta":471411,"kb_avail":754422300,"statfs":{"total":773094113280,"available":772528435200,"internally_reserved":0,"allocated":79687680,"data_stored":41331018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":482725433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":451,"apply_latency_ms":451,"commit_latency_ns":451000000,"apply_latency_ns":451000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.392619"},"pg_stats":[{"pgid":"20.8","version":"0'0","reported_seq":833,"reported_epoch":892,"state":"active+clean","last_fresh":"2023-10-30T12:39:36.676450+0000","last_change":"2023-10-30T12:19:35.856437+0000","last_active":"2023-10-30T12:39:36.676450+0000","last_peered":"2023-10-30T12:39:36.676450+0000","last_clean":"2023-10-30T12:39:36.676450+0000","last_became_active":"2023-10-30T12:19:35.850666+0000","last_became_peered":"2023-10-30T12:19:35.850666+0000","last_unstale":"2023-10-30T12:39:36.676450+0000","last_undegraded":"2023-10-30T12:39:36.676450+0000","last_fullsized":"2023-10-30T12:39:36.676450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:24.452037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":833,"reported_epoch":892,"state":"active+clean","last_fresh":"2023-10-30T12:39:36.679469+0000","last_change":"2023-10-30T12:19:38.560827+0000","last_active":"2023-10-30T12:39:36.679469+0000","last_peered":"2023-10-30T12:39:36.679469+0000","last_clean":"2023-10-30T12:39:36.679469+0000","last_became_active":"2023-10-30T12:19:38.546595+0000","last_became_peered":"2023-10-30T12:19:38.546595+0000","last_unstale":"2023-10-30T12:39:36.679469+0000","last_undegraded":"2023-10-30T12:39:36.679469+0000","last_fullsized":"2023-10-30T12:39:36.679469+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:22.315389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":833,"reported_epoch":892,"state":"active+clean","last_fresh":"2023-10-30T12:39:36.630077+0000","last_change":"2023-10-30T12:19:35.881177+0000","last_active":"2023-10-30T12:39:36.630077+0000","last_peered":"2023-10-30T12:39:36.630077+0000","last_clean":"2023-10-30T12:39:36.630077+0000","last_became_active":"2023-10-30T12:19:35.871000+0000","last_became_peered":"2023-10-30T12:19:35.871000+0000","last_unstale":"2023-10-30T12:39:36.630077+0000","last_undegraded":"2023-10-30T12:39:36.630077+0000","last_fullsized":"2023-10-30T12:39:36.630077+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:16.108884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":840,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:39.015181+0000","last_change":"2023-10-30T12:19:29.602055+0000","last_active":"2023-10-30T12:39:39.015181+0000","last_peered":"2023-10-30T12:39:39.015181+0000","last_clean":"2023-10-30T12:39:39.015181+0000","last_became_active":"2023-10-30T12:19:29.594505+0000","last_became_peered":"2023-10-30T12:19:29.594505+0000","last_unstale":"2023-10-30T12:39:39.015181+0000","last_undegraded":"2023-10-30T12:39:39.015181+0000","last_fullsized":"2023-10-30T12:39:39.015181+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:15:25.873922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":847,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.256860+0000","last_change":"2023-10-30T12:19:27.523347+0000","last_active":"2023-10-30T12:39:41.256860+0000","last_peered":"2023-10-30T12:39:41.256860+0000","last_clean":"2023-10-30T12:39:41.256860+0000","last_became_active":"2023-10-30T12:19:27.515739+0000","last_became_peered":"2023-10-30T12:19:27.515739+0000","last_unstale":"2023-10-30T12:39:41.256860+0000","last_undegraded":"2023-10-30T12:39:41.256860+0000","last_fullsized":"2023-10-30T12:39:41.256860+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:36.852443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":835,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.922009+0000","last_change":"2023-10-30T12:19:31.914796+0000","last_active":"2023-10-30T12:39:41.922009+0000","last_peered":"2023-10-30T12:39:41.922009+0000","last_clean":"2023-10-30T12:39:41.922009+0000","last_became_active":"2023-10-30T12:19:31.894071+0000","last_became_peered":"2023-10-30T12:19:31.894071+0000","last_unstale":"2023-10-30T12:39:41.922009+0000","last_undegraded":"2023-10-30T12:39:41.922009+0000","last_fullsized":"2023-10-30T12:39:41.922009+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:22.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":833,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.176984+0000","last_change":"2023-10-30T12:19:28.154596+0000","last_active":"2023-10-30T12:39:40.176984+0000","last_peered":"2023-10-30T12:39:40.176984+0000","last_clean":"2023-10-30T12:39:40.176984+0000","last_became_active":"2023-10-30T12:19:28.142591+0000","last_became_peered":"2023-10-30T12:19:28.142591+0000","last_unstale":"2023-10-30T12:39:40.176984+0000","last_undegraded":"2023-10-30T12:39:40.176984+0000","last_fullsized":"2023-10-30T12:39:40.176984+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:46:05.070599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":829,"reported_epoch":892,"state":"active+clean","last_fresh":"2023-10-30T12:39:36.661372+0000","last_change":"2023-10-30T12:19:38.653024+0000","last_active":"2023-10-30T12:39:36.661372+0000","last_peered":"2023-10-30T12:39:36.661372+0000","last_clean":"2023-10-30T12:39:36.661372+0000","last_became_active":"2023-10-30T12:19:38.643834+0000","last_became_peered":"2023-10-30T12:19:38.643834+0000","last_unstale":"2023-10-30T12:39:36.661372+0000","last_undegraded":"2023-10-30T12:39:36.661372+0000","last_fullsized":"2023-10-30T12:39:36.661372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:54.052947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":841,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.989491+0000","last_change":"2023-10-30T12:19:29.499316+0000","last_active":"2023-10-30T12:39:38.989491+0000","last_peered":"2023-10-30T12:39:38.989491+0000","last_clean":"2023-10-30T12:39:38.989491+0000","last_became_active":"2023-10-30T12:19:29.492043+0000","last_became_peered":"2023-10-30T12:19:29.492043+0000","last_unstale":"2023-10-30T12:39:38.989491+0000","last_undegraded":"2023-10-30T12:39:38.989491+0000","last_fullsized":"2023-10-30T12:39:38.989491+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:49.202008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1","version":"38'7","reported_seq":840,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.453703+0000","last_change":"2023-10-30T12:19:25.218622+0000","last_active":"2023-10-30T12:39:40.453703+0000","last_peered":"2023-10-30T12:39:40.453703+0000","last_clean":"2023-10-30T12:39:40.453703+0000","last_became_active":"2023-10-30T12:19:25.212397+0000","last_became_peered":"2023-10-30T12:19:25.212397+0000","last_unstale":"2023-10-30T12:39:40.453703+0000","last_undegraded":"2023-10-30T12:39:40.453703+0000","last_fullsized":"2023-10-30T12:39:40.453703+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:21.606857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":845,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.046753+0000","last_change":"2023-10-30T12:19:29.500965+0000","last_active":"2023-10-30T12:39:38.046753+0000","last_peered":"2023-10-30T12:39:38.046753+0000","last_clean":"2023-10-30T12:39:38.046753+0000","last_became_active":"2023-10-30T12:19:29.491706+0000","last_became_peered":"2023-10-30T12:19:29.491706+0000","last_unstale":"2023-10-30T12:39:38.046753+0000","last_undegraded":"2023-10-30T12:39:38.046753+0000","last_fullsized":"2023-10-30T12:39:38.046753+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:49.491590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":848,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.236376+0000","last_change":"2023-10-30T12:19:32.994118+0000","last_active":"2023-10-30T12:39:41.236376+0000","last_peered":"2023-10-30T12:39:41.236376+0000","last_clean":"2023-10-30T12:39:41.236376+0000","last_became_active":"2023-10-30T12:19:32.988808+0000","last_became_peered":"2023-10-30T12:19:32.988808+0000","last_unstale":"2023-10-30T12:39:41.236376+0000","last_undegraded":"2023-10-30T12:39:41.236376+0000","last_fullsized":"2023-10-30T12:39:41.236376+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:25.605041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":847,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.234304+0000","last_change":"2023-10-30T12:19:28.922326+0000","last_active":"2023-10-30T12:39:41.234304+0000","last_peered":"2023-10-30T12:39:41.234304+0000","last_clean":"2023-10-30T12:39:41.234304+0000","last_became_active":"2023-10-30T12:19:28.916380+0000","last_became_peered":"2023-10-30T12:19:28.916380+0000","last_unstale":"2023-10-30T12:39:41.234304+0000","last_undegraded":"2023-10-30T12:39:41.234304+0000","last_fullsized":"2023-10-30T12:39:41.234304+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:38:19.847301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":851,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.624076+0000","last_change":"2023-10-30T12:18:01.574170+0000","last_active":"2023-10-30T12:39:40.624076+0000","last_peered":"2023-10-30T12:39:40.624076+0000","last_clean":"2023-10-30T12:39:40.624076+0000","last_became_active":"2023-10-30T12:18:01.568787+0000","last_became_peered":"2023-10-30T12:18:01.568787+0000","last_unstale":"2023-10-30T12:39:40.624076+0000","last_undegraded":"2023-10-30T12:39:40.624076+0000","last_fullsized":"2023-10-30T12:39:40.624076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:12.243846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":833,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.947342+0000","last_change":"2023-10-30T12:19:40.949818+0000","last_active":"2023-10-30T12:39:38.947342+0000","last_peered":"2023-10-30T12:39:38.947342+0000","last_clean":"2023-10-30T12:39:38.947342+0000","last_became_active":"2023-10-30T12:19:40.940463+0000","last_became_peered":"2023-10-30T12:19:40.940463+0000","last_unstale":"2023-10-30T12:39:38.947342+0000","last_undegraded":"2023-10-30T12:39:38.947342+0000","last_fullsized":"2023-10-30T12:39:38.947342+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:43:05.246911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":839,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.954352+0000","last_change":"2023-10-30T12:18:01.632325+0000","last_active":"2023-10-30T12:39:38.954352+0000","last_peered":"2023-10-30T12:39:38.954352+0000","last_clean":"2023-10-30T12:39:38.954352+0000","last_became_active":"2023-10-30T12:18:01.602190+0000","last_became_peered":"2023-10-30T12:18:01.602190+0000","last_unstale":"2023-10-30T12:39:38.954352+0000","last_undegraded":"2023-10-30T12:39:38.954352+0000","last_fullsized":"2023-10-30T12:39:38.954352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:29:50.132707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":844,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.051655+0000","last_change":"2023-10-30T12:19:40.722588+0000","last_active":"2023-10-30T12:39:38.051655+0000","last_peered":"2023-10-30T12:39:38.051655+0000","last_clean":"2023-10-30T12:39:38.051655+0000","last_became_active":"2023-10-30T12:19:40.713186+0000","last_became_peered":"2023-10-30T12:19:40.713186+0000","last_unstale":"2023-10-30T12:39:38.051655+0000","last_undegraded":"2023-10-30T12:39:38.051655+0000","last_fullsized":"2023-10-30T12:39:38.051655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:38:31.344804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":850,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.071121+0000","last_change":"2023-10-30T12:18:01.060414+0000","last_active":"2023-10-30T12:39:38.071121+0000","last_peered":"2023-10-30T12:39:38.071121+0000","last_clean":"2023-10-30T12:39:38.071121+0000","last_became_active":"2023-10-30T12:18:01.051697+0000","last_became_peered":"2023-10-30T12:18:01.051697+0000","last_unstale":"2023-10-30T12:39:38.071121+0000","last_undegraded":"2023-10-30T12:39:38.071121+0000","last_fullsized":"2023-10-30T12:39:38.071121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:10.773828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":826,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.413791+0000","last_change":"2023-10-30T12:19:40.870027+0000","last_active":"2023-10-30T12:39:40.413791+0000","last_peered":"2023-10-30T12:39:40.413791+0000","last_clean":"2023-10-30T12:39:40.413791+0000","last_became_active":"2023-10-30T12:19:40.864175+0000","last_became_peered":"2023-10-30T12:19:40.864175+0000","last_unstale":"2023-10-30T12:39:40.413791+0000","last_undegraded":"2023-10-30T12:39:40.413791+0000","last_fullsized":"2023-10-30T12:39:40.413791+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:47.649334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":841,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.300250+0000","last_change":"2023-10-30T12:18:01.550154+0000","last_active":"2023-10-30T12:39:40.300250+0000","last_peered":"2023-10-30T12:39:40.300250+0000","last_clean":"2023-10-30T12:39:40.300250+0000","last_became_active":"2023-10-30T12:18:01.523269+0000","last_became_peered":"2023-10-30T12:18:01.523269+0000","last_unstale":"2023-10-30T12:39:40.300250+0000","last_undegraded":"2023-10-30T12:39:40.300250+0000","last_fullsized":"2023-10-30T12:39:40.300250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:08.405234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":834,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.298056+0000","last_change":"2023-10-30T12:19:40.449659+0000","last_active":"2023-10-30T12:39:40.298056+0000","last_peered":"2023-10-30T12:39:40.298056+0000","last_clean":"2023-10-30T12:39:40.298056+0000","last_became_active":"2023-10-30T12:19:40.440094+0000","last_became_peered":"2023-10-30T12:19:40.440094+0000","last_unstale":"2023-10-30T12:39:40.298056+0000","last_undegraded":"2023-10-30T12:39:40.298056+0000","last_fullsized":"2023-10-30T12:39:40.298056+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:12.412748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":851,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.048394+0000","last_change":"2023-10-30T12:18:01.509923+0000","last_active":"2023-10-30T12:39:38.048394+0000","last_peered":"2023-10-30T12:39:38.048394+0000","last_clean":"2023-10-30T12:39:38.048394+0000","last_became_active":"2023-10-30T12:18:01.504543+0000","last_became_peered":"2023-10-30T12:18:01.504543+0000","last_unstale":"2023-10-30T12:39:38.048394+0000","last_undegraded":"2023-10-30T12:39:38.048394+0000","last_fullsized":"2023-10-30T12:39:38.048394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:39.886008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":835,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:39.032349+0000","last_change":"2023-10-30T12:19:29.552208+0000","last_active":"2023-10-30T12:39:39.032349+0000","last_peered":"2023-10-30T12:39:39.032349+0000","last_clean":"2023-10-30T12:39:39.032349+0000","last_became_active":"2023-10-30T12:19:29.543275+0000","last_became_peered":"2023-10-30T12:19:29.543275+0000","last_unstale":"2023-10-30T12:39:39.032349+0000","last_undegraded":"2023-10-30T12:39:39.032349+0000","last_fullsized":"2023-10-30T12:39:39.032349+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:51.785658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":854,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.205204+0000","last_change":"2023-10-30T12:18:00.932461+0000","last_active":"2023-10-30T12:39:41.205204+0000","last_peered":"2023-10-30T12:39:41.205204+0000","last_clean":"2023-10-30T12:39:41.205204+0000","last_became_active":"2023-10-30T12:18:00.904525+0000","last_became_peered":"2023-10-30T12:18:00.904525+0000","last_unstale":"2023-10-30T12:39:41.205204+0000","last_undegraded":"2023-10-30T12:39:41.205204+0000","last_fullsized":"2023-10-30T12:39:41.205204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:42.734208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":827,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.858779+0000","last_change":"2023-10-30T12:19:26.068263+0000","last_active":"2023-10-30T12:39:41.858779+0000","last_peered":"2023-10-30T12:39:41.858779+0000","last_clean":"2023-10-30T12:39:41.858779+0000","last_became_active":"2023-10-30T12:19:26.061689+0000","last_became_peered":"2023-10-30T12:19:26.061689+0000","last_unstale":"2023-10-30T12:39:41.858779+0000","last_undegraded":"2023-10-30T12:39:41.858779+0000","last_fullsized":"2023-10-30T12:39:41.858779+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:50.946689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":851,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.014253+0000","last_change":"2023-10-30T12:18:01.030843+0000","last_active":"2023-10-30T12:39:38.014253+0000","last_peered":"2023-10-30T12:39:38.014253+0000","last_clean":"2023-10-30T12:39:38.014253+0000","last_became_active":"2023-10-30T12:18:01.003382+0000","last_became_peered":"2023-10-30T12:18:01.003382+0000","last_unstale":"2023-10-30T12:39:38.014253+0000","last_undegraded":"2023-10-30T12:39:38.014253+0000","last_fullsized":"2023-10-30T12:39:38.014253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:20.711334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":826,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.521943+0000","last_change":"2023-10-30T12:19:25.299105+0000","last_active":"2023-10-30T12:39:40.521943+0000","last_peered":"2023-10-30T12:39:40.521943+0000","last_clean":"2023-10-30T12:39:40.521943+0000","last_became_active":"2023-10-30T12:19:25.272220+0000","last_became_peered":"2023-10-30T12:19:25.272220+0000","last_unstale":"2023-10-30T12:39:40.521943+0000","last_undegraded":"2023-10-30T12:39:40.521943+0000","last_fullsized":"2023-10-30T12:39:40.521943+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:17.556977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":851,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.056001+0000","last_change":"2023-10-30T12:18:01.406267+0000","last_active":"2023-10-30T12:39:38.056001+0000","last_peered":"2023-10-30T12:39:38.056001+0000","last_clean":"2023-10-30T12:39:38.056001+0000","last_became_active":"2023-10-30T12:18:01.393697+0000","last_became_peered":"2023-10-30T12:18:01.393697+0000","last_unstale":"2023-10-30T12:39:38.056001+0000","last_undegraded":"2023-10-30T12:39:38.056001+0000","last_fullsized":"2023-10-30T12:39:38.056001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:15:29.710122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":941,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.832524+0000","last_change":"2023-10-30T12:17:56.604947+0000","last_active":"2023-10-30T12:39:41.832524+0000","last_peered":"2023-10-30T12:39:41.832524+0000","last_clean":"2023-10-30T12:39:41.832524+0000","last_became_active":"2023-10-30T12:17:56.575838+0000","last_became_peered":"2023-10-30T12:17:56.575838+0000","last_unstale":"2023-10-30T12:39:41.832524+0000","last_undegraded":"2023-10-30T12:39:41.832524+0000","last_fullsized":"2023-10-30T12:39:41.832524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_clean_scrub_stamp":"2023-10-30T12:17:53.592563+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:47.623228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":845,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.080335+0000","last_change":"2023-10-30T12:19:25.736207+0000","last_active":"2023-10-30T12:39:38.080335+0000","last_peered":"2023-10-30T12:39:38.080335+0000","last_clean":"2023-10-30T12:39:38.080335+0000","last_became_active":"2023-10-30T12:19:25.715885+0000","last_became_peered":"2023-10-30T12:19:25.715885+0000","last_unstale":"2023-10-30T12:39:38.080335+0000","last_undegraded":"2023-10-30T12:39:38.080335+0000","last_fullsized":"2023-10-30T12:39:38.080335+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:50.517927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":838,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:39.037645+0000","last_change":"2023-10-30T12:19:29.928614+0000","last_active":"2023-10-30T12:39:39.037645+0000","last_peered":"2023-10-30T12:39:39.037645+0000","last_clean":"2023-10-30T12:39:39.037645+0000","last_became_active":"2023-10-30T12:19:29.879226+0000","last_became_peered":"2023-10-30T12:19:29.879226+0000","last_unstale":"2023-10-30T12:39:39.037645+0000","last_undegraded":"2023-10-30T12:39:39.037645+0000","last_fullsized":"2023-10-30T12:39:39.037645+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:26:19.751528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":843,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.068307+0000","last_change":"2023-10-30T12:19:40.799718+0000","last_active":"2023-10-30T12:39:38.068307+0000","last_peered":"2023-10-30T12:39:38.068307+0000","last_clean":"2023-10-30T12:39:38.068307+0000","last_became_active":"2023-10-30T12:19:40.794341+0000","last_became_peered":"2023-10-30T12:19:40.794341+0000","last_unstale":"2023-10-30T12:39:38.068307+0000","last_undegraded":"2023-10-30T12:39:38.068307+0000","last_fullsized":"2023-10-30T12:39:38.068307+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:58.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":844,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.077563+0000","last_change":"2023-10-30T12:19:28.247323+0000","last_active":"2023-10-30T12:39:38.077563+0000","last_peered":"2023-10-30T12:39:38.077563+0000","last_clean":"2023-10-30T12:39:38.077563+0000","last_became_active":"2023-10-30T12:19:28.238513+0000","last_became_peered":"2023-10-30T12:19:28.238513+0000","last_unstale":"2023-10-30T12:39:38.077563+0000","last_undegraded":"2023-10-30T12:39:38.077563+0000","last_fullsized":"2023-10-30T12:39:38.077563+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:43.483993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":833,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.958239+0000","last_change":"2023-10-30T12:19:27.303856+0000","last_active":"2023-10-30T12:39:38.958239+0000","last_peered":"2023-10-30T12:39:38.958239+0000","last_clean":"2023-10-30T12:39:38.958239+0000","last_became_active":"2023-10-30T12:19:27.284772+0000","last_became_peered":"2023-10-30T12:19:27.284772+0000","last_unstale":"2023-10-30T12:39:38.958239+0000","last_undegraded":"2023-10-30T12:39:38.958239+0000","last_fullsized":"2023-10-30T12:39:38.958239+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:40.340431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":827,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.834340+0000","last_change":"2023-10-30T12:19:33.492307+0000","last_active":"2023-10-30T12:39:41.834340+0000","last_peered":"2023-10-30T12:39:41.834340+0000","last_clean":"2023-10-30T12:39:41.834340+0000","last_became_active":"2023-10-30T12:19:33.483122+0000","last_became_peered":"2023-10-30T12:19:33.483122+0000","last_unstale":"2023-10-30T12:39:41.834340+0000","last_undegraded":"2023-10-30T12:39:41.834340+0000","last_fullsized":"2023-10-30T12:39:41.834340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:16.210880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":839,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.998483+0000","last_change":"2023-10-30T12:19:26.678316+0000","last_active":"2023-10-30T12:39:38.998483+0000","last_peered":"2023-10-30T12:39:38.998483+0000","last_clean":"2023-10-30T12:39:38.998483+0000","last_became_active":"2023-10-30T12:19:26.663005+0000","last_became_peered":"2023-10-30T12:19:26.663005+0000","last_unstale":"2023-10-30T12:39:38.998483+0000","last_undegraded":"2023-10-30T12:39:38.998483+0000","last_fullsized":"2023-10-30T12:39:38.998483+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:22:45.623416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":844,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.011668+0000","last_change":"2023-10-30T12:19:26.773408+0000","last_active":"2023-10-30T12:39:38.011668+0000","last_peered":"2023-10-30T12:39:38.011668+0000","last_clean":"2023-10-30T12:39:38.011668+0000","last_became_active":"2023-10-30T12:19:26.766234+0000","last_became_peered":"2023-10-30T12:19:26.766234+0000","last_unstale":"2023-10-30T12:39:38.011668+0000","last_undegraded":"2023-10-30T12:39:38.011668+0000","last_fullsized":"2023-10-30T12:39:38.011668+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:39.619411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":830,"reported_epoch":892,"state":"active+clean","last_fresh":"2023-10-30T12:39:36.659758+0000","last_change":"2023-10-30T12:19:29.219323+0000","last_active":"2023-10-30T12:39:36.659758+0000","last_peered":"2023-10-30T12:39:36.659758+0000","last_clean":"2023-10-30T12:39:36.659758+0000","last_became_active":"2023-10-30T12:19:29.191499+0000","last_became_peered":"2023-10-30T12:19:29.191499+0000","last_unstale":"2023-10-30T12:39:36.659758+0000","last_undegraded":"2023-10-30T12:39:36.659758+0000","last_fullsized":"2023-10-30T12:39:36.659758+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.259618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":848,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.270235+0000","last_change":"2023-10-30T12:19:30.420980+0000","last_active":"2023-10-30T12:39:41.270235+0000","last_peered":"2023-10-30T12:39:41.270235+0000","last_clean":"2023-10-30T12:39:41.270235+0000","last_became_active":"2023-10-30T12:19:30.408141+0000","last_became_peered":"2023-10-30T12:19:30.408141+0000","last_unstale":"2023-10-30T12:39:41.270235+0000","last_undegraded":"2023-10-30T12:39:41.270235+0000","last_fullsized":"2023-10-30T12:39:41.270235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:04:46.816451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":834,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.223908+0000","last_change":"2023-10-30T12:19:40.539149+0000","last_active":"2023-10-30T12:39:40.223908+0000","last_peered":"2023-10-30T12:39:40.223908+0000","last_clean":"2023-10-30T12:39:40.223908+0000","last_became_active":"2023-10-30T12:19:40.528642+0000","last_became_peered":"2023-10-30T12:19:40.528642+0000","last_unstale":"2023-10-30T12:39:40.223908+0000","last_undegraded":"2023-10-30T12:39:40.223908+0000","last_fullsized":"2023-10-30T12:39:40.223908+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:34.653320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":835,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.295847+0000","last_change":"2023-10-30T12:19:30.748646+0000","last_active":"2023-10-30T12:39:40.295847+0000","last_peered":"2023-10-30T12:39:40.295847+0000","last_clean":"2023-10-30T12:39:40.295847+0000","last_became_active":"2023-10-30T12:19:30.720900+0000","last_became_peered":"2023-10-30T12:19:30.720900+0000","last_unstale":"2023-10-30T12:39:40.295847+0000","last_undegraded":"2023-10-30T12:39:40.295847+0000","last_fullsized":"2023-10-30T12:39:40.295847+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:59.574207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509705,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63332,"kb_used_data":9304,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308508,"statfs":{"total":96636764160,"available":96571912192,"internally_reserved":0,"allocated":9527296,"data_stored":4823141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":23.231999999999999,"5min":17.655999999999999,"15min":16.648},"min":{"1min":6.4530000000000003,"5min":4.0250000000000004,"15min":4.0250000000000004},"max":{"1min":181.905,"5min":233.52500000000001,"15min":233.52500000000001},"last":30.998999999999999},{"interface":"front","average":{"1min":27.143000000000001,"5min":21.59,"15min":22.366},"min":{"1min":9.7650000000000006,"5min":5.6740000000000004,"15min":5.6740000000000004},"max":{"1min":183.072,"5min":214.31899999999999,"15min":365.47399999999999},"last":35.723999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.035,"5min":12.183,"15min":14.696},"min":{"1min":6.0800000000000001,"5min":4.4359999999999999,"15min":4.2389999999999999},"max":{"1min":63.286999999999999,"5min":63.286999999999999,"15min":142.51400000000001},"last":6.6500000000000004},{"interface":"front","average":{"1min":17.978999999999999,"5min":14.823,"15min":17.242000000000001},"min":{"1min":6.056,"5min":5.1689999999999996,"15min":5.1669999999999998},"max":{"1min":61.552,"5min":112.276,"15min":211.11600000000001},"last":7.8440000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.715,"5min":21.622,"15min":20.561},"min":{"1min":6.5259999999999998,"5min":6.2530000000000001,"15min":6.2530000000000001},"max":{"1min":61.927,"5min":187.00399999999999,"15min":187.00399999999999},"last":9.4510000000000005},{"interface":"front","average":{"1min":20.795999999999999,"5min":23.681999999999999,"15min":21.565000000000001},"min":{"1min":8.2710000000000008,"5min":6.5359999999999996,"15min":5.6260000000000003},"max":{"1min":80.686999999999998,"5min":240.99799999999999,"15min":240.99799999999999},"last":8.8460000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":20.48,"5min":26.420000000000002,"15min":25.207999999999998},"min":{"1min":10.49,"5min":7.5990000000000002,"15min":6.2240000000000002},"max":{"1min":62.517000000000003,"5min":279.94099999999997,"15min":279.94099999999997},"last":12.83},{"interface":"front","average":{"1min":17.039000000000001,"5min":24.077999999999999,"15min":23.585000000000001},"min":{"1min":8.1370000000000005,"5min":6.617,"15min":6.5960000000000001},"max":{"1min":55.351999999999997,"5min":180.74600000000001,"15min":180.74600000000001},"last":36.521999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":13.237,"5min":19.042000000000002,"15min":18.274999999999999},"min":{"1min":6.617,"5min":6.617,"15min":6.1840000000000002},"max":{"1min":59.103000000000002,"5min":184.88,"15min":184.88},"last":31.678000000000001},{"interface":"front","average":{"1min":20.768999999999998,"5min":26.350000000000001,"15min":25.687999999999999},"min":{"1min":11.442,"5min":9.4160000000000004,"15min":7.3540000000000001},"max":{"1min":54.426000000000002,"5min":374.38900000000001,"15min":374.38900000000001},"last":13.537000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":19.882000000000001,"5min":25.530999999999999,"15min":25.718},"min":{"1min":10.507,"5min":8.1210000000000004,"15min":6.2729999999999997},"max":{"1min":43.252000000000002,"5min":218.28800000000001,"15min":218.28800000000001},"last":39.472999999999999},{"interface":"front","average":{"1min":17.363,"5min":25.651,"15min":23.794},"min":{"1min":6.7160000000000002,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":34.281999999999996,"5min":195.74700000000001,"15min":195.74700000000001},"last":32.414999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":21.611999999999998,"5min":23.324000000000002,"15min":27.591999999999999},"min":{"1min":9.8620000000000001,"5min":9.593,"15min":8.3320000000000007},"max":{"1min":69.704999999999998,"5min":178.68100000000001,"15min":224.71000000000001},"last":65.272000000000006},{"interface":"front","average":{"1min":19.506,"5min":24.533000000000001,"15min":25.074000000000002},"min":{"1min":9.7799999999999994,"5min":7.2469999999999999,"15min":7.2469999999999999},"max":{"1min":39.325000000000003,"5min":187.14500000000001,"15min":210.05199999999999},"last":33.552999999999997}]}]},{"osd":5,"up_from":15,"seq":64424509705,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82020,"kb_used_data":10132,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289820,"statfs":{"total":96636764160,"available":96552775680,"internally_reserved":0,"allocated":10375168,"data_stored":5538149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":82,"apply_latency_ms":82,"commit_latency_ns":82000000,"apply_latency_ns":82000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:38:43 2023","interfaces":[{"interface":"back","average":{"1min":22.297000000000001,"5min":15.888999999999999,"15min":28.812000000000001},"min":{"1min":4.4119999999999999,"5min":4.1050000000000004,"15min":4.1050000000000004},"max":{"1min":136.36000000000001,"5min":158.87799999999999,"15min":627.18200000000002},"last":12.56},{"interface":"front","average":{"1min":18.611999999999998,"5min":16.850000000000001,"15min":26.960999999999999},"min":{"1min":6.367,"5min":4.7469999999999999,"15min":4.7469999999999999},"max":{"1min":94.358999999999995,"5min":146.374,"15min":341.17000000000002},"last":7.75}]},{"osd":1,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":18.443999999999999,"5min":16.146000000000001,"15min":28.567},"min":{"1min":5.8040000000000003,"5min":5.5300000000000002,"15min":5.5300000000000002},"max":{"1min":138.964,"5min":138.964,"15min":620.21500000000003},"last":4.9160000000000004},{"interface":"front","average":{"1min":17.212,"5min":17.853000000000002,"15min":27.408000000000001},"min":{"1min":7.1970000000000001,"5min":5.6150000000000002,"15min":5.4109999999999996},"max":{"1min":93.744,"5min":162.25200000000001,"15min":334.62},"last":13.255000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":18.535,"5min":18.928000000000001,"15min":23.741},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":134.77000000000001,"5min":208.28100000000001,"15min":246.37},"last":8.5500000000000007},{"interface":"front","average":{"1min":21.039000000000001,"5min":20.257999999999999,"15min":24.73},"min":{"1min":8.4540000000000006,"5min":6.3049999999999997,"15min":5.2770000000000001},"max":{"1min":141.99799999999999,"5min":141.99799999999999,"15min":348.41000000000003},"last":13.888}]},{"osd":3,"last update":"Mon Oct 30 12:39:26 2023","interfaces":[{"interface":"back","average":{"1min":19.34,"5min":23.516999999999999,"15min":25.564},"min":{"1min":6.8339999999999996,"5min":6.3860000000000001,"15min":6.3860000000000001},"max":{"1min":97.822999999999993,"5min":148.125,"15min":273.10300000000001},"last":16.361999999999998},{"interface":"front","average":{"1min":24.561,"5min":21.800000000000001,"15min":24.858000000000001},"min":{"1min":9.3970000000000002,"5min":6.2889999999999997,"15min":5.8029999999999999},"max":{"1min":94.296000000000006,"5min":154.857,"15min":282.28500000000003},"last":14.513}]},{"osd":4,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":25.477,"5min":23.068000000000001,"15min":33.439},"min":{"1min":11.792999999999999,"5min":9.5109999999999992,"15min":7.0259999999999998},"max":{"1min":144.102,"5min":144.102,"15min":512.01199999999994},"last":15.071999999999999},{"interface":"front","average":{"1min":27.440999999999999,"5min":25.501000000000001,"15min":31.102},"min":{"1min":7.4530000000000003,"5min":7.4530000000000003,"15min":6.3890000000000002},"max":{"1min":113.208,"5min":169.45500000000001,"15min":386.62200000000001},"last":9.8640000000000008}]},{"osd":6,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":28.742000000000001,"5min":28.922000000000001,"15min":31.957000000000001},"min":{"1min":14.225,"5min":11.138,"15min":7.9729999999999999},"max":{"1min":144.726,"5min":242.16800000000001,"15min":588.93899999999996},"last":15.701000000000001},{"interface":"front","average":{"1min":32.106999999999999,"5min":34.963999999999999,"15min":37.119},"min":{"1min":16.084,"5min":12.661,"15min":7.25},"max":{"1min":161.18000000000001,"5min":319.17700000000002,"15min":500.173},"last":17.097999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:39:26 2023","interfaces":[{"interface":"back","average":{"1min":20.66,"5min":27.561,"15min":28.265999999999998},"min":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"max":{"1min":113.35899999999999,"5min":258.61200000000002,"15min":258.61200000000002},"last":10.956},{"interface":"front","average":{"1min":16.838999999999999,"5min":22.385999999999999,"15min":24.34},"min":{"1min":6.109,"5min":6.109,"15min":6.109},"max":{"1min":55.759,"5min":145.15000000000001,"15min":245.71899999999999},"last":11.893000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509705,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63564,"kb_used_data":9512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308276,"statfs":{"total":96636764160,"available":96571674624,"internally_reserved":0,"allocated":9740288,"data_stored":4978430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":8.8330000000000002,"5min":15.872,"15min":22.242999999999999},"min":{"1min":4.7569999999999997,"5min":4.2140000000000004,"15min":4.0620000000000003},"max":{"1min":16.516999999999999,"5min":158.499,"15min":295.56400000000002},"last":8.7710000000000008},{"interface":"front","average":{"1min":8.3000000000000007,"5min":18.66,"15min":23.565999999999999},"min":{"1min":4.4080000000000004,"5min":4.4080000000000004,"15min":4.4080000000000004},"max":{"1min":14.081,"5min":182.96600000000001,"15min":368.01900000000001},"last":7.1189999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:38:45 2023","interfaces":[{"interface":"back","average":{"1min":11.939,"5min":17.212,"15min":23.172000000000001},"min":{"1min":5.9829999999999997,"5min":5.6420000000000003,"15min":4.5389999999999997},"max":{"1min":53.116,"5min":171.24100000000001,"15min":322.97800000000001},"last":12.673},{"interface":"front","average":{"1min":17.463999999999999,"5min":20.486999999999998,"15min":22.018000000000001},"min":{"1min":6.9050000000000002,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":105.38800000000001,"5min":168.833,"15min":245.44900000000001},"last":18.204000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:38:51 2023","interfaces":[{"interface":"back","average":{"1min":18.649999999999999,"5min":26.581,"15min":25.079000000000001},"min":{"1min":7.9740000000000002,"5min":6.0039999999999996,"15min":6.0039999999999996},"max":{"1min":112.88,"5min":316.74799999999999,"15min":316.74799999999999},"last":8.2379999999999995},{"interface":"front","average":{"1min":16.074999999999999,"5min":26.199999999999999,"15min":26.724},"min":{"1min":8.0690000000000008,"5min":5.9489999999999998,"15min":5.9489999999999998},"max":{"1min":53.756,"5min":317.55399999999997,"15min":317.55399999999997},"last":14.481999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":14.321999999999999,"5min":23.728999999999999,"15min":28.404},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":6.7270000000000003},"max":{"1min":21.91,"5min":161.197,"15min":383.87099999999998},"last":10.694000000000001},{"interface":"front","average":{"1min":13.757999999999999,"5min":26.026,"15min":26.616},"min":{"1min":8.5,"5min":8.5,"15min":6.6529999999999996},"max":{"1min":18.109999999999999,"5min":260.94200000000001,"15min":284.61700000000002},"last":16.137}]},{"osd":5,"last update":"Mon Oct 30 12:38:45 2023","interfaces":[{"interface":"back","average":{"1min":23.707999999999998,"5min":27.972000000000001,"15min":30.349},"min":{"1min":7.9429999999999996,"5min":7.9429999999999996,"15min":5.7119999999999997},"max":{"1min":183.55000000000001,"5min":185.47200000000001,"15min":261.80200000000002},"last":19.524999999999999},{"interface":"front","average":{"1min":17.798999999999999,"5min":22.954999999999998,"15min":29.309000000000001},"min":{"1min":9.1099999999999994,"5min":6.3760000000000003,"15min":6.3760000000000003},"max":{"1min":102.428,"5min":174.38999999999999,"15min":371.35700000000003},"last":20.347000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":23.914999999999999,"5min":27.789999999999999,"15min":27.952999999999999},"min":{"1min":10.460000000000001,"5min":9.4939999999999998,"15min":8.0060000000000002},"max":{"1min":95.795000000000002,"5min":215.31800000000001,"15min":312.44999999999999},"last":16.774999999999999},{"interface":"front","average":{"1min":19.725999999999999,"5min":28.399999999999999,"15min":27.664000000000001},"min":{"1min":9.859,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":63.609999999999999,"5min":266.50599999999997,"15min":320.50999999999999},"last":11.413}]},{"osd":7,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":17.331,"5min":26.408000000000001,"15min":28.349},"min":{"1min":9.048,"5min":9.048,"15min":7.9370000000000003},"max":{"1min":55.015000000000001,"5min":193.95400000000001,"15min":319.84199999999998},"last":20.981000000000002},{"interface":"front","average":{"1min":15.836,"5min":26.617000000000001,"15min":27.077000000000002},"min":{"1min":10.515000000000001,"5min":8.9489999999999998,"15min":8.5410000000000004},"max":{"1min":28.527000000000001,"5min":252.232,"15min":315.06900000000002},"last":13.680999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509703,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85592,"kb_used_data":9768,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286248,"statfs":{"total":96636764160,"available":96549117952,"internally_reserved":0,"allocated":10002432,"data_stored":5188384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":82,"apply_latency_ms":82,"commit_latency_ns":82000000,"apply_latency_ns":82000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.731999999999999,"5min":21.190000000000001,"15min":21.754999999999999},"min":{"1min":4.609,"5min":4.2460000000000004,"15min":4.1159999999999997},"max":{"1min":210.398,"5min":303.58600000000001,"15min":303.58600000000001},"last":6.9340000000000002},{"interface":"front","average":{"1min":26.295999999999999,"5min":17.559999999999999,"15min":21.213000000000001},"min":{"1min":6.0049999999999999,"5min":4.8819999999999997,"15min":4.1070000000000002},"max":{"1min":187.82400000000001,"5min":187.82400000000001,"15min":268.68799999999999},"last":13.012}]},{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":42.889000000000003,"5min":25.684999999999999,"15min":23.533999999999999},"min":{"1min":7.7990000000000004,"5min":5.6070000000000002,"15min":4.8940000000000001},"max":{"1min":220.96600000000001,"5min":220.96600000000001,"15min":389.303},"last":12.266999999999999},{"interface":"front","average":{"1min":43.094999999999999,"5min":25.721,"15min":24.445},"min":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":5.0890000000000004},"max":{"1min":222.328,"5min":222.328,"15min":468.13299999999998},"last":6.2850000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":19.759,"5min":21.931000000000001,"15min":22.350999999999999},"min":{"1min":8.0250000000000004,"5min":6.8230000000000004,"15min":5.6150000000000002},"max":{"1min":135.97399999999999,"5min":352.09399999999999,"15min":352.09399999999999},"last":9.6430000000000007},{"interface":"front","average":{"1min":18.385000000000002,"5min":20.425999999999998,"15min":23.646999999999998},"min":{"1min":7.7889999999999997,"5min":6.9850000000000003,"15min":6.9850000000000003},"max":{"1min":94.552999999999997,"5min":184.04599999999999,"15min":287.33699999999999},"last":17.280999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.673000000000002,"5min":25.640000000000001,"15min":28.405000000000001},"min":{"1min":8.6929999999999996,"5min":7.1799999999999997,"15min":6.5010000000000003},"max":{"1min":211.03200000000001,"5min":310.12099999999998,"15min":310.12099999999998},"last":8.2880000000000003},{"interface":"front","average":{"1min":31.611000000000001,"5min":25.715,"15min":25.268000000000001},"min":{"1min":9.2690000000000001,"5min":6.7169999999999996,"15min":6.0919999999999996},"max":{"1min":187.15700000000001,"5min":359.87799999999999,"15min":359.87799999999999},"last":8.9420000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":44.430999999999997,"5min":29.558,"15min":30.042999999999999},"min":{"1min":9.8840000000000003,"5min":9,"15min":6.851},"max":{"1min":274.15300000000002,"5min":360.88099999999997,"15min":564.26700000000005},"last":10.279},{"interface":"front","average":{"1min":50.723999999999997,"5min":28.849,"15min":28.324999999999999},"min":{"1min":10.619,"5min":8.8680000000000003,"15min":8.0109999999999992},"max":{"1min":289.68099999999998,"5min":289.68099999999998,"15min":294.392},"last":13.667999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":30.876999999999999,"5min":29.062999999999999,"15min":30.353999999999999},"min":{"1min":11.016,"5min":9.3369999999999997,"15min":8.8149999999999995},"max":{"1min":197.09200000000001,"5min":374.33999999999997,"15min":374.33999999999997},"last":14.846},{"interface":"front","average":{"1min":33.301000000000002,"5min":31.440000000000001,"15min":31.552},"min":{"1min":11.416,"5min":11.416,"15min":8.9079999999999995},"max":{"1min":177.63499999999999,"5min":352.79500000000002,"15min":352.79500000000002},"last":15.577}]},{"osd":7,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":36.420000000000002,"5min":32.140999999999998,"15min":34.933},"min":{"1min":12.868,"5min":11.401999999999999,"15min":8.5099999999999998},"max":{"1min":203.70099999999999,"5min":382.68299999999999,"15min":382.68299999999999},"last":16.646000000000001},{"interface":"front","average":{"1min":24.998999999999999,"5min":29.645,"15min":32.134},"min":{"1min":11.976000000000001,"5min":9.0850000000000009,"15min":8.7699999999999996},"max":{"1min":86.024000000000001,"5min":361.47199999999998,"15min":562.05700000000002},"last":18.204999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509703,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67412,"kb_used_data":9868,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94304428,"statfs":{"total":96636764160,"available":96567734272,"internally_reserved":0,"allocated":10104832,"data_stored":5216115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:02 2023","interfaces":[{"interface":"back","average":{"1min":9.0120000000000005,"5min":15.948,"15min":19.835000000000001},"min":{"1min":4.4260000000000002,"5min":4.4260000000000002,"15min":3.8079999999999998},"max":{"1min":20.417000000000002,"5min":121.476,"15min":337.86700000000002},"last":5.9729999999999999},{"interface":"front","average":{"1min":10.103,"5min":14.271000000000001,"15min":18.513000000000002},"min":{"1min":5.6310000000000002,"5min":4.4279999999999999,"15min":4.1399999999999997},"max":{"1min":18.503,"5min":119.87,"15min":343.983},"last":11.227}]},{"osd":1,"last update":"Mon Oct 30 12:38:53 2023","interfaces":[{"interface":"back","average":{"1min":10.884,"5min":16.09,"15min":23.937000000000001},"min":{"1min":7.0860000000000003,"5min":4.9790000000000001,"15min":4.6559999999999997},"max":{"1min":16.172000000000001,"5min":153.751,"15min":501.19900000000001},"last":12.455},{"interface":"front","average":{"1min":13.513999999999999,"5min":18.212,"15min":22.693999999999999},"min":{"1min":6.9859999999999998,"5min":6.9859999999999998,"15min":6.649},"max":{"1min":34.274000000000001,"5min":122.767,"15min":391.58300000000003},"last":6.9180000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:38:57 2023","interfaces":[{"interface":"back","average":{"1min":13.710000000000001,"5min":21.341999999999999,"15min":27.303000000000001},"min":{"1min":7.7140000000000004,"5min":6.923,"15min":6.6379999999999999},"max":{"1min":34.911999999999999,"5min":173.56899999999999,"15min":414.69299999999998},"last":8.9190000000000005},{"interface":"front","average":{"1min":14.257,"5min":21.814,"15min":28.373000000000001},"min":{"1min":7.484,"5min":7.1769999999999996,"15min":6.6269999999999998},"max":{"1min":32.726999999999997,"5min":197.904,"15min":502.19299999999998},"last":13.968}]},{"osd":4,"last update":"Mon Oct 30 12:38:41 2023","interfaces":[{"interface":"back","average":{"1min":17.321000000000002,"5min":22.783999999999999,"15min":27.199000000000002},"min":{"1min":12.071999999999999,"5min":8.0540000000000003,"15min":6.415},"max":{"1min":36.734999999999999,"5min":157.77199999999999,"15min":331.834},"last":8},{"interface":"front","average":{"1min":10.712999999999999,"5min":16.815999999999999,"15min":20.263999999999999},"min":{"1min":6.5099999999999998,"5min":5.984,"15min":5.984},"max":{"1min":19.161000000000001,"5min":134.452,"15min":179.89099999999999},"last":10.460000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:38:42 2023","interfaces":[{"interface":"back","average":{"1min":16.175000000000001,"5min":25.510999999999999,"15min":34.662999999999997},"min":{"1min":8.6720000000000006,"5min":8.0619999999999994,"15min":7.4500000000000002},"max":{"1min":36.591000000000001,"5min":245.22800000000001,"15min":695.71100000000001},"last":17.966000000000001},{"interface":"front","average":{"1min":18.864999999999998,"5min":27.582000000000001,"15min":36.226999999999997},"min":{"1min":12.657,"5min":9.6349999999999998,"15min":9.3930000000000007},"max":{"1min":37.334000000000003,"5min":241.41999999999999,"15min":578.82500000000005},"last":9.7189999999999994}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":16.959,"5min":19.983000000000001,"15min":20.140000000000001},"min":{"1min":8.1110000000000007,"5min":6.3239999999999998,"15min":6.1100000000000003},"max":{"1min":49.598999999999997,"5min":146.75,"15min":182.97499999999999},"last":11.861000000000001},{"interface":"front","average":{"1min":20.119,"5min":23.864000000000001,"15min":25.939},"min":{"1min":11.311,"5min":9.8819999999999997,"15min":8.7729999999999997},"max":{"1min":45.713000000000001,"5min":154.619,"15min":315.71100000000001},"last":15.045}]},{"osd":7,"last update":"Mon Oct 30 12:38:53 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":18.695,"15min":22.042999999999999},"min":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"max":{"1min":27.41,"5min":123.926,"15min":228.67400000000001},"last":18.713000000000001},{"interface":"front","average":{"1min":19.202000000000002,"5min":25.521000000000001,"15min":29.041},"min":{"1min":11.582000000000001,"5min":10.462999999999999,"15min":9.0589999999999993},"max":{"1min":38.158000000000001,"5min":160.93600000000001,"15min":328.51799999999997},"last":19.545000000000002}]}]},{"osd":1,"up_from":15,"seq":64424509704,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63800,"kb_used_data":10184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308040,"statfs":{"total":96636764160,"available":96571432960,"internally_reserved":0,"allocated":10428416,"data_stored":5595488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":10.207000000000001,"5min":11.294,"15min":18.631},"min":{"1min":5.5519999999999996,"5min":4.3760000000000003,"15min":4.3760000000000003},"max":{"1min":24.184999999999999,"5min":75.006,"15min":294.67099999999999},"last":14.157},{"interface":"front","average":{"1min":11.981999999999999,"5min":11.672000000000001,"15min":17.254000000000001},"min":{"1min":6.4210000000000003,"5min":5.2400000000000002,"15min":4.952},"max":{"1min":27.530999999999999,"5min":61.932000000000002,"15min":241.066},"last":7.1820000000000004}]},{"osd":2,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":16.41,"5min":14.955,"15min":19.184000000000001},"min":{"1min":6.4299999999999997,"5min":6.4299999999999997,"15min":6.0519999999999996},"max":{"1min":67.140000000000001,"5min":75.491,"15min":295.50599999999997},"last":8.3179999999999996},{"interface":"front","average":{"1min":18.949999999999999,"5min":13.51,"15min":19.122},"min":{"1min":6.8810000000000002,"5min":6.1580000000000004,"15min":5.327},"max":{"1min":135.00800000000001,"5min":135.00800000000001,"15min":269.99200000000002},"last":11.167999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":19.390000000000001,"15min":21.367999999999999},"min":{"1min":6.9889999999999999,"5min":6.4779999999999998,"15min":6.4779999999999998},"max":{"1min":25.774999999999999,"5min":310.40800000000002,"15min":551.58000000000004},"last":9.5630000000000006},{"interface":"front","average":{"1min":17.298999999999999,"5min":28.062000000000001,"15min":27.957000000000001},"min":{"1min":9.8819999999999997,"5min":9.8819999999999997,"15min":9.1999999999999993},"max":{"1min":29.579000000000001,"5min":338.88600000000002,"15min":562.06899999999996},"last":15.07}]},{"osd":4,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":14.186,"5min":16.617000000000001,"15min":21.454000000000001},"min":{"1min":7.0469999999999997,"5min":6.7480000000000002,"15min":6.6109999999999998},"max":{"1min":40.465000000000003,"5min":84.212000000000003,"15min":285.26499999999999},"last":20.274000000000001},{"interface":"front","average":{"1min":14.521000000000001,"5min":16.108000000000001,"15min":22.126000000000001},"min":{"1min":9.0500000000000007,"5min":6.7450000000000001,"15min":6.7450000000000001},"max":{"1min":42.087000000000003,"5min":62.616,"15min":327.59699999999998},"last":16.891999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":35.115000000000002,"5min":25.553999999999998,"15min":26.443999999999999},"min":{"1min":9.9410000000000007,"5min":8.1639999999999997,"15min":7.3070000000000004},"max":{"1min":322.59899999999999,"5min":322.59899999999999,"15min":322.59899999999999},"last":16.023},{"interface":"front","average":{"1min":32.095999999999997,"5min":21.616,"15min":24.370999999999999},"min":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"max":{"1min":283.06,"5min":283.06,"15min":286.43799999999999},"last":12.630000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":13.723000000000001,"5min":17.039000000000001,"15min":22.716000000000001},"min":{"1min":7.6059999999999999,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":28.151,"5min":71.662999999999997,"15min":348.91000000000003},"last":10.596},{"interface":"front","average":{"1min":18.815999999999999,"5min":23.582999999999998,"15min":26.449000000000002},"min":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":8.4819999999999993},"max":{"1min":33.466000000000001,"5min":99.231999999999999,"15min":301.99799999999999},"last":19.335000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":21.850999999999999,"5min":21.949000000000002,"15min":25.800999999999998},"min":{"1min":13.108000000000001,"5min":9.2780000000000005,"15min":7.8739999999999997},"max":{"1min":68.974999999999994,"5min":107.703,"15min":305.19900000000001},"last":17.609000000000002},{"interface":"front","average":{"1min":20.207000000000001,"5min":20.809000000000001,"15min":26.821000000000002},"min":{"1min":8.8640000000000008,"5min":8.8640000000000008,"15min":8.0920000000000005},"max":{"1min":70.406999999999996,"5min":106.678,"15min":277.05200000000002},"last":11.855}]}]},{"osd":0,"up_from":15,"seq":64424509704,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63140,"kb_used_data":9544,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308700,"statfs":{"total":96636764160,"available":96572108800,"internally_reserved":0,"allocated":9773056,"data_stored":5007107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":14.013,"5min":16.620999999999999,"15min":19.768000000000001},"min":{"1min":6.117,"5min":4.1799999999999997,"15min":4.1799999999999997},"max":{"1min":73.113,"5min":232.798,"15min":287.096},"last":47.450000000000003},{"interface":"front","average":{"1min":11.015000000000001,"5min":14.513,"15min":22.515999999999998},"min":{"1min":7.6879999999999997,"5min":5.2370000000000001,"15min":4.7590000000000003},"max":{"1min":29.097000000000001,"5min":218.61099999999999,"15min":425.762},"last":44.664999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:39:40 2023","interfaces":[{"interface":"back","average":{"1min":21.314,"5min":17.907,"15min":20.555},"min":{"1min":8.3599999999999994,"5min":5.2480000000000002,"15min":5.2480000000000002},"max":{"1min":78.947999999999993,"5min":228.90799999999999,"15min":299.46600000000001},"last":78.947999999999993},{"interface":"front","average":{"1min":28.640000000000001,"5min":18.164000000000001,"15min":20.786000000000001},"min":{"1min":6.7779999999999996,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":113.76000000000001,"5min":138.13499999999999,"15min":258.45400000000001},"last":87.332999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.376999999999999,"5min":16.640999999999998,"15min":20.597999999999999},"min":{"1min":9.6329999999999991,"5min":6.5499999999999998,"15min":5.2110000000000003},"max":{"1min":41.456000000000003,"5min":166.21600000000001,"15min":264.97199999999998},"last":55.999000000000002},{"interface":"front","average":{"1min":19.736000000000001,"5min":21.084,"15min":25.367000000000001},"min":{"1min":8.4079999999999995,"5min":8.4079999999999995,"15min":7.218},"max":{"1min":54.859000000000002,"5min":230.73699999999999,"15min":258.93200000000002},"last":58.859000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6620000000000008,"5min":18.376000000000001,"15min":22.655999999999999},"min":{"1min":5.681,"5min":5.681,"15min":5.681},"max":{"1min":24.405999999999999,"5min":209.64599999999999,"15min":209.64599999999999},"last":61.680999999999997},{"interface":"front","average":{"1min":20.053999999999998,"5min":25.309000000000001,"15min":23.466999999999999},"min":{"1min":8.8819999999999997,"5min":8.0990000000000002,"15min":5.8499999999999996},"max":{"1min":90.787999999999997,"5min":208.92400000000001,"15min":208.92400000000001},"last":52.835999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.98,"5min":29.106000000000002,"15min":29.109999999999999},"min":{"1min":10.965,"5min":8.9440000000000008,"15min":7.5510000000000002},"max":{"1min":31.939,"5min":245.16300000000001,"15min":285.57100000000003},"last":60.929000000000002},{"interface":"front","average":{"1min":17.673999999999999,"5min":21.956,"15min":28.129999999999999},"min":{"1min":10.518000000000001,"5min":7.2720000000000002,"15min":7.2720000000000002},"max":{"1min":30.361000000000001,"5min":142.25,"15min":292.07900000000001},"last":54.409999999999997}]},{"osd":6,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":15.827999999999999,"5min":20.135999999999999,"15min":21.259},"min":{"1min":7.7000000000000002,"5min":7.4370000000000003,"15min":7.4370000000000003},"max":{"1min":43.283999999999999,"5min":171.99600000000001,"15min":236.40000000000001},"last":60.557000000000002},{"interface":"front","average":{"1min":18.968,"5min":24.116,"15min":25.454999999999998},"min":{"1min":12.252000000000001,"5min":10.470000000000001,"15min":8.9199999999999999},"max":{"1min":31.385000000000002,"5min":178.59999999999999,"15min":220.68700000000001},"last":65.087000000000003}]},{"osd":7,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":19.300999999999998,"5min":20.318000000000001,"15min":24.036000000000001},"min":{"1min":11.853,"5min":9.2100000000000009,"15min":9.2100000000000009},"max":{"1min":32.664999999999999,"5min":143.49600000000001,"15min":245.87100000000001},"last":55.232999999999997},{"interface":"front","average":{"1min":16.231000000000002,"5min":22.702000000000002,"15min":24.879999999999999},"min":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":32.033000000000001,"5min":181.15100000000001,"15min":260.04000000000002},"last":63.960999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575112,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63560,"kb_used_data":9508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308280,"statfs":{"total":96636764160,"available":96571678720,"internally_reserved":0,"allocated":9736192,"data_stored":4984204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":9.4269999999999996,"5min":10.32,"15min":22.806999999999999},"min":{"1min":4.0960000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"max":{"1min":23.024000000000001,"5min":99.075000000000003,"15min":623.43700000000001},"last":6.4779999999999998},{"interface":"front","average":{"1min":10.904,"5min":11.211,"15min":21.835000000000001},"min":{"1min":4.7279999999999998,"5min":4.202,"15min":4.202},"max":{"1min":24.524999999999999,"5min":51.253999999999998,"15min":563.08900000000006},"last":7.6710000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":11.772,"15min":19.108000000000001},"min":{"1min":6.04,"5min":5.0019999999999998,"15min":5.0019999999999998},"max":{"1min":13.172000000000001,"5min":68.641999999999996,"15min":276.80399999999997},"last":7.2800000000000002},{"interface":"front","average":{"1min":13.029999999999999,"5min":16.312999999999999,"15min":24.184000000000001},"min":{"1min":9.3659999999999997,"5min":7.008,"15min":7.008},"max":{"1min":18.454000000000001,"5min":80.891999999999996,"15min":289.52800000000002},"last":13.73}]},{"osd":2,"last update":"Mon Oct 30 12:38:51 2023","interfaces":[{"interface":"back","average":{"1min":11.6,"5min":18.939,"15min":22.053999999999998},"min":{"1min":6.1749999999999998,"5min":6.1749999999999998,"15min":5.0529999999999999},"max":{"1min":20.262,"5min":329.81999999999999,"15min":329.81999999999999},"last":10.012},{"interface":"front","average":{"1min":13.555999999999999,"5min":17.236000000000001,"15min":22.600000000000001},"min":{"1min":7.5229999999999997,"5min":6.8129999999999997,"15min":6.0350000000000001},"max":{"1min":18.695,"5min":132.25800000000001,"15min":245.63},"last":15.695}]},{"osd":3,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":14.976000000000001,"5min":19.353999999999999,"15min":25.904},"min":{"1min":8.6329999999999991,"5min":6.4950000000000001,"15min":6.4950000000000001},"max":{"1min":33.936,"5min":97.286000000000001,"15min":185.98400000000001},"last":16.648},{"interface":"front","average":{"1min":17.303000000000001,"5min":21.863,"15min":28.611999999999998},"min":{"1min":8.2799999999999994,"5min":7.7000000000000002,"15min":6.8159999999999998},"max":{"1min":61.064,"5min":247.65899999999999,"15min":361.625},"last":12.743}]},{"osd":4,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":13.592000000000001,"5min":20.183,"15min":29.582000000000001},"min":{"1min":8.7720000000000002,"5min":5.8150000000000004,"15min":5.8150000000000004},"max":{"1min":17.824000000000002,"5min":117.967,"15min":838.78700000000003},"last":17.553000000000001},{"interface":"front","average":{"1min":12.276,"5min":16.821000000000002,"15min":27.617000000000001},"min":{"1min":6.9329999999999998,"5min":6.9329999999999998,"15min":6.6280000000000001},"max":{"1min":18.846,"5min":82.543999999999997,"15min":836.30899999999997},"last":10.99}]},{"osd":5,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":20.395,"5min":22.149000000000001,"15min":32.493000000000002},"min":{"1min":9.3989999999999991,"5min":7.1440000000000001,"15min":6.5110000000000001},"max":{"1min":64.566000000000003,"5min":186.54499999999999,"15min":504.84100000000001},"last":11.638},{"interface":"front","average":{"1min":24.806999999999999,"5min":22.687999999999999,"15min":32.274000000000001},"min":{"1min":11.058999999999999,"5min":7.5919999999999996,"15min":6.7249999999999996},"max":{"1min":108.858,"5min":185.61099999999999,"15min":405.85000000000002},"last":8.6980000000000004}]},{"osd":7,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":16.547999999999998,"5min":21.190999999999999,"15min":28.661999999999999},"min":{"1min":10.946,"5min":9.3940000000000001,"15min":7.5579999999999998},"max":{"1min":20.657,"5min":99.480999999999995,"15min":402.315},"last":18.251999999999999},{"interface":"front","average":{"1min":17.120000000000001,"5min":24.227,"15min":27.654},"min":{"1min":13.159000000000001,"5min":8.8019999999999996,"15min":8.8019999999999996},"max":{"1min":21.486999999999998,"5min":264.99299999999999,"15min":291.69},"last":14.337999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:39:44.521 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:39:44.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.621+0000 7fb934dc2640 10 monclient: get_monmap_and_config 2023-10-30T12:39:44.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.621+0000 7fb934dc2640 10 monclient: build_initial_monmap 2023-10-30T12:39:44.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.621+0000 7fb934dc2640 10 monclient: monmap: 2023-10-30T12:39:44.623 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:44.624 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:44.624 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:44.622524+0000 2023-10-30T12:39:44.624 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:44.622524+0000 2023-10-30T12:39:44.624 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:44.624 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:44.625 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:44.625 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:44.625 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:44.625 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:44.625 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:44.626 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 1 Processor -- start 2023-10-30T12:39:44.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 1 -- start start 2023-10-30T12:39:44.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 10 monclient: init 2023-10-30T12:39:44.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:44.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-10-30T12:39:44.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9300b7c10 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:44.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb930141940 0x7fb930141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 10 monclient(hunting): picked mon.noname-a con 0x7fb930141940 addr v2:172.21.15.148:3300/0 2023-10-30T12:39:44.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb93013e730 0x7fb93013eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 10 monclient(hunting): picked mon.noname-c con 0x7fb93013e730 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:44.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9300a9350 con 0x7fb9300b7c10 2023-10-30T12:39:44.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:44.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9300670a0 con 0x7fb930141940 2023-10-30T12:39:44.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:44.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9300b7ff0 con 0x7fb93013e730 2023-10-30T12:39:44.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:44.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.622+0000 7fb934dc2640 10 monclient(hunting): authenticate will time out at 2876.827776s 2023-10-30T12:39:44.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb927fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb930141940 0x7fb930141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb92ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb93013e730 0x7fb93013eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb927fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb930141940 0x7fb930141d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55146/0 (socket says 172.21.15.148:55146) 2023-10-30T12:39:44.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb927fff640 1 -- 172.21.15.148:0/3112059461 learned_addr learned my addr 172.21.15.148:0/3112059461 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:44.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb927fff640 10 monclient(hunting): get_auth_request con 0x7fb930141940 auth_method 0 2023-10-30T12:39:44.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb927fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:44.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb927fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:44.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb927fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:44.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb92ed9d640 10 monclient(hunting): get_auth_request con 0x7fb93013e730 auth_method 0 2023-10-30T12:39:44.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb92ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:44.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb92ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:44.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.623+0000 7fb92ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:44.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.624+0000 7fb927fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:44.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.624+0000 7fb927fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:44.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.624+0000 7fb927fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:44.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.624+0000 7fb92ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:44.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.624+0000 7fb92ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:44.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.624+0000 7fb92ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:44.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.625+0000 7fb92ed9d640 10 monclient(hunting): handle_auth_done global_id 22048 payload 274 2023-10-30T12:39:44.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.625+0000 7fb92ed9d640 1 -- 172.21.15.148:0/3112059461 >> v2:172.21.15.148:3300/0 conn(0x7fb930141940 msgr2=0x7fb930141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:44.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.625+0000 7fb92ed9d640 1 --2- 172.21.15.148:0/3112059461 >> v2:172.21.15.148:3300/0 conn(0x7fb930141940 0x7fb930141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.625+0000 7fb92ed9d640 1 -- 172.21.15.148:0/3112059461 >> v1:172.21.15.148:6789/0 conn(0x7fb9300b7c10 legacy=0x7fb9300a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:44.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.625+0000 7fb92ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:39:44.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.625+0000 7fb92ed9d640 1 monclient: found mon.noname-c 2023-10-30T12:39:44.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.625+0000 7fb92ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:44.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.625+0000 7fb92ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:39:44.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.625+0000 7fb92ed9d640 1 -- 172.21.15.148:0/3112059461 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9300aa6a0 con 0x7fb93013e730 2023-10-30T12:39:44.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.627+0000 7fb92ed9d640 1 --2- 172.21.15.148:0/3112059461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93013e730 0x7fb93013eb80 secure :-1 s=READY pgs=5431 cs=0 l=1 rev1=1 crypto rx=0x7fb9300b37f0 tx=0x7fb92802bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd2b395fccdd84fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:44.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 1 -- 172.21.15.148:0/3112059461 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92802cb60 con 0x7fb93013e730 2023-10-30T12:39:44.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:44.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:44.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 10 monclient: dump: 2023-10-30T12:39:44.642 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:44.642 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:44.642 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.642 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.643 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:44.643 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:44.643 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:44.643 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:44.643 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:44.644 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 10 monclient: _finish_auth 0 2023-10-30T12:39:44.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 10 monclient: _check_auth_tickets 2023-10-30T12:39:44.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:44.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 1 -- 172.21.15.148:0/3112059461 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92802ccd0 con 0x7fb93013e730 2023-10-30T12:39:44.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:44.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 1 -- 172.21.15.148:0/3112059461 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb928034660 con 0x7fb93013e730 2023-10-30T12:39:44.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb934dc2640 5 monclient: authenticate success, global_id 22048 2023-10-30T12:39:44.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb934dc2640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:44.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb934dc2640 4 set_mon_vals no callback set 2023-10-30T12:39:44.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:44.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:44.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.631+0000 7fb92dd9b640 10 monclient: dump: 2023-10-30T12:39:44.646 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:44.646 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:44.646 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.647 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.647 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:44.647 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:44.647 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:44.648 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:44.648 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:44.648 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.632+0000 7fb934dc2640 10 monclient: shutdown 2023-10-30T12:39:44.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.632+0000 7fb934dc2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:44.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.632+0000 7fb934dc2640 1 -- 172.21.15.148:0/3112059461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93013e730 msgr2=0x7fb93013eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:44.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.632+0000 7fb934dc2640 1 --2- 172.21.15.148:0/3112059461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93013e730 0x7fb93013eb80 secure :-1 s=READY pgs=5431 cs=0 l=1 rev1=1 crypto rx=0x7fb9300b37f0 tx=0x7fb92802bdb0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.632+0000 7fb934dc2640 1 -- 172.21.15.148:0/3112059461 shutdown_connections 2023-10-30T12:39:44.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.632+0000 7fb934dc2640 1 --2- 172.21.15.148:0/3112059461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93013e730 0x7fb93013eb80 unknown :-1 s=CLOSED pgs=5431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.632+0000 7fb934dc2640 1 --2- 172.21.15.148:0/3112059461 >> v2:172.21.15.148:3300/0 conn(0x7fb930141940 0x7fb930141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.632+0000 7fb934dc2640 1 -- 172.21.15.148:0/3112059461 >> 172.21.15.148:0/3112059461 conn(0x7fb93006c660 msgr2=0x7fb9300b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:44.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.632+0000 7fb934dc2640 1 -- 172.21.15.148:0/3112059461 shutdown_connections 2023-10-30T12:39:44.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.632+0000 7fb934dc2640 1 -- 172.21.15.148:0/3112059461 wait complete. 2023-10-30T12:39:44.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.633+0000 7fb934dc2640 10 monclient: build_initial_monmap 2023-10-30T12:39:44.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.633+0000 7fb934dc2640 10 monclient: monmap: 2023-10-30T12:39:44.650 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:44.651 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:44.651 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:44.651 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:44.651 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:44.651 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:44.652 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:44.652 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:44.652 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:44.652 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.633+0000 7fb934dc2640 1 librados: starting msgr at 2023-10-30T12:39:44.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.633+0000 7fb934dc2640 1 librados: starting objecter 2023-10-30T12:39:44.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.633+0000 7fb934dc2640 1 Processor -- start 2023-10-30T12:39:44.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.633+0000 7fb934dc2640 1 -- start start 2023-10-30T12:39:44.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.633+0000 7fb934dc2640 1 librados: setting wanted keys 2023-10-30T12:39:44.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.633+0000 7fb934dc2640 1 librados: calling monclient init 2023-10-30T12:39:44.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.633+0000 7fb934dc2640 10 monclient: init 2023-10-30T12:39:44.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:44.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:39:44.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9300b7c10 0x7fb930139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 10 monclient(hunting): picked mon.noname-b con 0x7fb9300b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:44.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb93013e730 0x7fb9301da400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 10 monclient(hunting): picked mon.noname-c con 0x7fb93013e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:44.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb930141940 0x7fb9300ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 10 monclient(hunting): picked mon.noname-a con 0x7fb930141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:44.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:44.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93013ee90 con 0x7fb930141940 2023-10-30T12:39:44.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:44.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300ad370 con 0x7fb9300b7c10 2023-10-30T12:39:44.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:44.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300b0d30 con 0x7fb93013e730 2023-10-30T12:39:44.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:44.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb934dc2640 10 monclient(hunting): authenticate will time out at 2876.839329s 2023-10-30T12:39:44.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb92e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9300b7c10 0x7fb930139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.634+0000 7fb92ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb930141940 0x7fb9300ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb927fff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb93013e730 0x7fb9301da400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9300b7c10 0x7fb930139200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:55486/0 (socket says 172.21.15.148:55486) 2023-10-30T12:39:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92e59c640 1 -- 172.21.15.148:0/3498940826 learned_addr learned my addr 172.21.15.148:0/3498940826 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92e59c640 10 monclient(hunting): get_auth_request con 0x7fb9300b7c10 auth_method 0 2023-10-30T12:39:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92ed9d640 10 monclient(hunting): get_auth_request con 0x7fb930141940 auth_method 0 2023-10-30T12:39:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb927fff640 10 monclient(hunting): get_auth_request con 0x7fb93013e730 auth_method 0 2023-10-30T12:39:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb927fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb927fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb927fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:44.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:44.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.635+0000 7fb92ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:44.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.636+0000 7fb927fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:44.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.636+0000 7fb927fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:44.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.636+0000 7fb927fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:44.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.638+0000 7fb92e59c640 10 monclient(hunting): handle_auth_done global_id 22051 payload 824 2023-10-30T12:39:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.638+0000 7fb92e59c640 1 -- 172.21.15.148:0/3498940826 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb93013e730 msgr2=0x7fb9301da400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.638+0000 7fb92e59c640 1 --2- 172.21.15.148:0/3498940826 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb93013e730 0x7fb9301da400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.638+0000 7fb92e59c640 1 -- 172.21.15.148:0/3498940826 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb930141940 msgr2=0x7fb9300ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.638+0000 7fb92e59c640 1 --2- 172.21.15.148:0/3498940826 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb930141940 0x7fb9300ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.638+0000 7fb92e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:39:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.638+0000 7fb92e59c640 1 monclient: found mon.noname-b 2023-10-30T12:39:44.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.638+0000 7fb92e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:44.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.638+0000 7fb92e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:39:44.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.638+0000 7fb92e59c640 1 -- 172.21.15.148:0/3498940826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9300aed00 con 0x7fb9300b7c10 2023-10-30T12:39:44.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.640+0000 7fb92e59c640 1 --2- 172.21.15.148:0/3498940826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9300b7c10 0x7fb930139200 secure :-1 s=READY pgs=5432 cs=0 l=1 rev1=1 crypto rx=0x7fb92000c410 tx=0x7fb92000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:44.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.640+0000 7fb926ffd640 10 client.?.objecter ms_handle_connect 0x7fb9300b7c10 2023-10-30T12:39:44.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.640+0000 7fb926ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:44.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 1 -- 172.21.15.148:0/3498940826 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9200035f0 con 0x7fb9300b7c10 2023-10-30T12:39:44.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:44.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:44.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 10 monclient: dump: 2023-10-30T12:39:44.666 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:44.666 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:44.666 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.667 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.667 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:44.667 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:44.667 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:44.667 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:44.667 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:44.668 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 10 monclient: _finish_auth 0 2023-10-30T12:39:44.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:39:44.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:44.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 1 -- 172.21.15.148:0/3498940826 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92000e040 con 0x7fb9300b7c10 2023-10-30T12:39:44.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb934dc2640 5 monclient: authenticate success, global_id 22051 2023-10-30T12:39:44.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb934dc2640 10 monclient: _renew_subs 2023-10-30T12:39:44.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb934dc2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:44.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb934dc2640 1 -- 172.21.15.148:0/3498940826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9301dab10 con 0x7fb9300b7c10 2023-10-30T12:39:44.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:44.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 1 -- 172.21.15.148:0/3498940826 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9200039e0 con 0x7fb9300b7c10 2023-10-30T12:39:44.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:44.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:44.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb926ffd640 10 monclient: dump: 2023-10-30T12:39:44.670 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:44.672 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:44.672 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:44.672 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:44.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb92d59a640 4 set_mon_vals no callback set 2023-10-30T12:39:44.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb934dc2640 10 client.22051.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:44.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb934dc2640 10 monclient: _renew_subs 2023-10-30T12:39:44.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb934dc2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:44.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb934dc2640 1 -- 172.21.15.148:0/3498940826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9301db030 con 0x7fb9300b7c10 2023-10-30T12:39:44.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.645+0000 7fb934dc2640 1 librados: init done 2023-10-30T12:39:44.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.646+0000 7fb934dc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:44.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.646+0000 7fb934dc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:44.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.646+0000 7fb934dc2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:44.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.646+0000 7fb934dc2640 1 -- 172.21.15.148:0/3498940826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9300a97c0 con 0x7fb9300b7c10 2023-10-30T12:39:44.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.667+0000 7fb926ffd640 1 -- 172.21.15.148:0/3498940826 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fb920003b50 con 0x7fb9300b7c10 2023-10-30T12:39:44.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.668+0000 7fb926ffd640 1 --2- 172.21.15.148:0/3498940826 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb91006c810 0x7fb91006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:44.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.668+0000 7fb927fff640 1 --2- 172.21.15.148:0/3498940826 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb91006c810 0x7fb91006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:44.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.668+0000 7fb927fff640 10 monclient: get_auth_request con 0x7fb91006c810 auth_method 0 2023-10-30T12:39:44.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.668+0000 7fb927fff640 1 --2- 172.21.15.148:0/3498940826 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb91006c810 0x7fb91006ecd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb9300af2c0 tx=0x7fb918009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:44.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.668+0000 7fb926ffd640 10 client.22051.objecter ms_handle_connect 0x7fb91006c810 2023-10-30T12:39:44.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.896+0000 7fb926ffd640 1 -- 172.21.15.148:0/3498940826 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(899..899 src has 501..899) v4 ==== 5133+0+0 (secure 0 0 0) 0x7fb9200010c0 con 0x7fb9300b7c10 2023-10-30T12:39:44.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.896+0000 7fb926ffd640 10 client.22051.objecter ms_dispatch 0x7fb930000b90 osd_map(899..899 src has 501..899) v4 2023-10-30T12:39:44.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.896+0000 7fb926ffd640 3 client.22051.objecter handle_osd_map got epochs [899,899] > 0 2023-10-30T12:39:44.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.896+0000 7fb926ffd640 3 client.22051.objecter handle_osd_map decoding full epoch 899 2023-10-30T12:39:44.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.896+0000 7fb926ffd640 20 client.22051.objecter dump_active .. 0 homeless 2023-10-30T12:39:44.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.906+0000 7fb926ffd640 1 -- 172.21.15.148:0/3498940826 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb920057730 con 0x7fb9300b7c10 2023-10-30T12:39:44.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.906+0000 7fb926ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:44.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:44.906+0000 7fb926ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:45.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.070+0000 7fb934dc2640 1 -- 172.21.15.148:0/3498940826 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9301422a0 con 0x7fb91006c810 2023-10-30T12:39:45.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.075+0000 7fb926ffd640 1 -- 172.21.15.148:0/3498940826 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151975 (secure 0 0 0) 0x7fb9301422a0 con 0x7fb91006c810 2023-10-30T12:39:45.078 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:39:45.079 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T12:39:45.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 10 librados: watch_flush enter 2023-10-30T12:39:45.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 10 librados: watch_flush exit 2023-10-30T12:39:45.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 20 client.22051.objecter shutdown clearing up homeless session... 2023-10-30T12:39:45.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 10 client.22051.objecter successfully canceled tick 2023-10-30T12:39:45.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 -- 172.21.15.148:0/3498940826 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb91006c810 msgr2=0x7fb91006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 --2- 172.21.15.148:0/3498940826 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb91006c810 0x7fb91006ecd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb9300af2c0 tx=0x7fb918009240 comp rx=0 tx=0).stop 2023-10-30T12:39:45.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 10 monclient: shutdown 2023-10-30T12:39:45.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:45.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 -- 172.21.15.148:0/3498940826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9300b7c10 msgr2=0x7fb930139200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 --2- 172.21.15.148:0/3498940826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9300b7c10 0x7fb930139200 secure :-1 s=READY pgs=5432 cs=0 l=1 rev1=1 crypto rx=0x7fb92000c410 tx=0x7fb92000c8d0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 -- 172.21.15.148:0/3498940826 shutdown_connections 2023-10-30T12:39:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 --2- 172.21.15.148:0/3498940826 >> v2:172.21.15.148:6800/3205090179 conn(0x7fb91006c810 0x7fb91006ecd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 --2- 172.21.15.148:0/3498940826 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb930141940 0x7fb9300ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 --2- 172.21.15.148:0/3498940826 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb93013e730 0x7fb9301da400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 --2- 172.21.15.148:0/3498940826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9300b7c10 0x7fb930139200 unknown :-1 s=CLOSED pgs=5432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 -- 172.21.15.148:0/3498940826 >> 172.21.15.148:0/3498940826 conn(0x7fb93006c660 msgr2=0x7fb9300738f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:45.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 -- 172.21.15.148:0/3498940826 shutdown_connections 2023-10-30T12:39:45.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 -- 172.21.15.148:0/3498940826 wait complete. 2023-10-30T12:39:45.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.077+0000 7fb934dc2640 1 librados: shutdown 2023-10-30T12:39:45.090 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1560,"stamp":"2023-10-30T12:39:43.390404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":552420,"kb_used_data":77820,"kb_used_omap":12,"kb_used_meta":471411,"kb_avail":754422300,"statfs":{"total":773094113280,"available":772528435200,"internally_reserved":0,"allocated":79687680,"data_stored":41331018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":482725433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":451,"apply_latency_ms":451,"commit_latency_ns":451000000,"apply_latency_ns":451000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.392619"},"pg_stats":[{"pgid":"20.8","version":"0'0","reported_seq":833,"reported_epoch":892,"state":"active+clean","last_fresh":"2023-10-30T12:39:36.676450+0000","last_change":"2023-10-30T12:19:35.856437+0000","last_active":"2023-10-30T12:39:36.676450+0000","last_peered":"2023-10-30T12:39:36.676450+0000","last_clean":"2023-10-30T12:39:36.676450+0000","last_became_active":"2023-10-30T12:19:35.850666+0000","last_became_peered":"2023-10-30T12:19:35.850666+0000","last_unstale":"2023-10-30T12:39:36.676450+0000","last_undegraded":"2023-10-30T12:39:36.676450+0000","last_fullsized":"2023-10-30T12:39:36.676450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:24.452037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":833,"reported_epoch":892,"state":"active+clean","last_fresh":"2023-10-30T12:39:36.679469+0000","last_change":"2023-10-30T12:19:38.560827+0000","last_active":"2023-10-30T12:39:36.679469+0000","last_peered":"2023-10-30T12:39:36.679469+0000","last_clean":"2023-10-30T12:39:36.679469+0000","last_became_active":"2023-10-30T12:19:38.546595+0000","last_became_peered":"2023-10-30T12:19:38.546595+0000","last_unstale":"2023-10-30T12:39:36.679469+0000","last_undegraded":"2023-10-30T12:39:36.679469+0000","last_fullsized":"2023-10-30T12:39:36.679469+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:22.315389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":833,"reported_epoch":892,"state":"active+clean","last_fresh":"2023-10-30T12:39:36.630077+0000","last_change":"2023-10-30T12:19:35.881177+0000","last_active":"2023-10-30T12:39:36.630077+0000","last_peered":"2023-10-30T12:39:36.630077+0000","last_clean":"2023-10-30T12:39:36.630077+0000","last_became_active":"2023-10-30T12:19:35.871000+0000","last_became_peered":"2023-10-30T12:19:35.871000+0000","last_unstale":"2023-10-30T12:39:36.630077+0000","last_undegraded":"2023-10-30T12:39:36.630077+0000","last_fullsized":"2023-10-30T12:39:36.630077+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:16.108884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":840,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:39.015181+0000","last_change":"2023-10-30T12:19:29.602055+0000","last_active":"2023-10-30T12:39:39.015181+0000","last_peered":"2023-10-30T12:39:39.015181+0000","last_clean":"2023-10-30T12:39:39.015181+0000","last_became_active":"2023-10-30T12:19:29.594505+0000","last_became_peered":"2023-10-30T12:19:29.594505+0000","last_unstale":"2023-10-30T12:39:39.015181+0000","last_undegraded":"2023-10-30T12:39:39.015181+0000","last_fullsized":"2023-10-30T12:39:39.015181+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:15:25.873922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":847,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.256860+0000","last_change":"2023-10-30T12:19:27.523347+0000","last_active":"2023-10-30T12:39:41.256860+0000","last_peered":"2023-10-30T12:39:41.256860+0000","last_clean":"2023-10-30T12:39:41.256860+0000","last_became_active":"2023-10-30T12:19:27.515739+0000","last_became_peered":"2023-10-30T12:19:27.515739+0000","last_unstale":"2023-10-30T12:39:41.256860+0000","last_undegraded":"2023-10-30T12:39:41.256860+0000","last_fullsized":"2023-10-30T12:39:41.256860+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:36.852443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":835,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.922009+0000","last_change":"2023-10-30T12:19:31.914796+0000","last_active":"2023-10-30T12:39:41.922009+0000","last_peered":"2023-10-30T12:39:41.922009+0000","last_clean":"2023-10-30T12:39:41.922009+0000","last_became_active":"2023-10-30T12:19:31.894071+0000","last_became_peered":"2023-10-30T12:19:31.894071+0000","last_unstale":"2023-10-30T12:39:41.922009+0000","last_undegraded":"2023-10-30T12:39:41.922009+0000","last_fullsized":"2023-10-30T12:39:41.922009+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:22.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":833,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.176984+0000","last_change":"2023-10-30T12:19:28.154596+0000","last_active":"2023-10-30T12:39:40.176984+0000","last_peered":"2023-10-30T12:39:40.176984+0000","last_clean":"2023-10-30T12:39:40.176984+0000","last_became_active":"2023-10-30T12:19:28.142591+0000","last_became_peered":"2023-10-30T12:19:28.142591+0000","last_unstale":"2023-10-30T12:39:40.176984+0000","last_undegraded":"2023-10-30T12:39:40.176984+0000","last_fullsized":"2023-10-30T12:39:40.176984+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:46:05.070599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":829,"reported_epoch":892,"state":"active+clean","last_fresh":"2023-10-30T12:39:36.661372+0000","last_change":"2023-10-30T12:19:38.653024+0000","last_active":"2023-10-30T12:39:36.661372+0000","last_peered":"2023-10-30T12:39:36.661372+0000","last_clean":"2023-10-30T12:39:36.661372+0000","last_became_active":"2023-10-30T12:19:38.643834+0000","last_became_peered":"2023-10-30T12:19:38.643834+0000","last_unstale":"2023-10-30T12:39:36.661372+0000","last_undegraded":"2023-10-30T12:39:36.661372+0000","last_fullsized":"2023-10-30T12:39:36.661372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:54.052947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":841,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.989491+0000","last_change":"2023-10-30T12:19:29.499316+0000","last_active":"2023-10-30T12:39:38.989491+0000","last_peered":"2023-10-30T12:39:38.989491+0000","last_clean":"2023-10-30T12:39:38.989491+0000","last_became_active":"2023-10-30T12:19:29.492043+0000","last_became_peered":"2023-10-30T12:19:29.492043+0000","last_unstale":"2023-10-30T12:39:38.989491+0000","last_undegraded":"2023-10-30T12:39:38.989491+0000","last_fullsized":"2023-10-30T12:39:38.989491+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:49.202008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1","version":"38'7","reported_seq":840,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.453703+0000","last_change":"2023-10-30T12:19:25.218622+0000","last_active":"2023-10-30T12:39:40.453703+0000","last_peered":"2023-10-30T12:39:40.453703+0000","last_clean":"2023-10-30T12:39:40.453703+0000","last_became_active":"2023-10-30T12:19:25.212397+0000","last_became_peered":"2023-10-30T12:19:25.212397+0000","last_unstale":"2023-10-30T12:39:40.453703+0000","last_undegraded":"2023-10-30T12:39:40.453703+0000","last_fullsized":"2023-10-30T12:39:40.453703+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:21.606857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":845,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.046753+0000","last_change":"2023-10-30T12:19:29.500965+0000","last_active":"2023-10-30T12:39:38.046753+0000","last_peered":"2023-10-30T12:39:38.046753+0000","last_clean":"2023-10-30T12:39:38.046753+0000","last_became_active":"2023-10-30T12:19:29.491706+0000","last_became_peered":"2023-10-30T12:19:29.491706+0000","last_unstale":"2023-10-30T12:39:38.046753+0000","last_undegraded":"2023-10-30T12:39:38.046753+0000","last_fullsized":"2023-10-30T12:39:38.046753+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:49.491590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":848,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.236376+0000","last_change":"2023-10-30T12:19:32.994118+0000","last_active":"2023-10-30T12:39:41.236376+0000","last_peered":"2023-10-30T12:39:41.236376+0000","last_clean":"2023-10-30T12:39:41.236376+0000","last_became_active":"2023-10-30T12:19:32.988808+0000","last_became_peered":"2023-10-30T12:19:32.988808+0000","last_unstale":"2023-10-30T12:39:41.236376+0000","last_undegraded":"2023-10-30T12:39:41.236376+0000","last_fullsized":"2023-10-30T12:39:41.236376+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:25.605041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":847,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.234304+0000","last_change":"2023-10-30T12:19:28.922326+0000","last_active":"2023-10-30T12:39:41.234304+0000","last_peered":"2023-10-30T12:39:41.234304+0000","last_clean":"2023-10-30T12:39:41.234304+0000","last_became_active":"2023-10-30T12:19:28.916380+0000","last_became_peered":"2023-10-30T12:19:28.916380+0000","last_unstale":"2023-10-30T12:39:41.234304+0000","last_undegraded":"2023-10-30T12:39:41.234304+0000","last_fullsized":"2023-10-30T12:39:41.234304+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:38:19.847301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":851,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.624076+0000","last_change":"2023-10-30T12:18:01.574170+0000","last_active":"2023-10-30T12:39:40.624076+0000","last_peered":"2023-10-30T12:39:40.624076+0000","last_clean":"2023-10-30T12:39:40.624076+0000","last_became_active":"2023-10-30T12:18:01.568787+0000","last_became_peered":"2023-10-30T12:18:01.568787+0000","last_unstale":"2023-10-30T12:39:40.624076+0000","last_undegraded":"2023-10-30T12:39:40.624076+0000","last_fullsized":"2023-10-30T12:39:40.624076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:12.243846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":833,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.947342+0000","last_change":"2023-10-30T12:19:40.949818+0000","last_active":"2023-10-30T12:39:38.947342+0000","last_peered":"2023-10-30T12:39:38.947342+0000","last_clean":"2023-10-30T12:39:38.947342+0000","last_became_active":"2023-10-30T12:19:40.940463+0000","last_became_peered":"2023-10-30T12:19:40.940463+0000","last_unstale":"2023-10-30T12:39:38.947342+0000","last_undegraded":"2023-10-30T12:39:38.947342+0000","last_fullsized":"2023-10-30T12:39:38.947342+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:43:05.246911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":839,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.954352+0000","last_change":"2023-10-30T12:18:01.632325+0000","last_active":"2023-10-30T12:39:38.954352+0000","last_peered":"2023-10-30T12:39:38.954352+0000","last_clean":"2023-10-30T12:39:38.954352+0000","last_became_active":"2023-10-30T12:18:01.602190+0000","last_became_peered":"2023-10-30T12:18:01.602190+0000","last_unstale":"2023-10-30T12:39:38.954352+0000","last_undegraded":"2023-10-30T12:39:38.954352+0000","last_fullsized":"2023-10-30T12:39:38.954352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:29:50.132707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":844,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.051655+0000","last_change":"2023-10-30T12:19:40.722588+0000","last_active":"2023-10-30T12:39:38.051655+0000","last_peered":"2023-10-30T12:39:38.051655+0000","last_clean":"2023-10-30T12:39:38.051655+0000","last_became_active":"2023-10-30T12:19:40.713186+0000","last_became_peered":"2023-10-30T12:19:40.713186+0000","last_unstale":"2023-10-30T12:39:38.051655+0000","last_undegraded":"2023-10-30T12:39:38.051655+0000","last_fullsized":"2023-10-30T12:39:38.051655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:38:31.344804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":850,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.071121+0000","last_change":"2023-10-30T12:18:01.060414+0000","last_active":"2023-10-30T12:39:38.071121+0000","last_peered":"2023-10-30T12:39:38.071121+0000","last_clean":"2023-10-30T12:39:38.071121+0000","last_became_active":"2023-10-30T12:18:01.051697+0000","last_became_peered":"2023-10-30T12:18:01.051697+0000","last_unstale":"2023-10-30T12:39:38.071121+0000","last_undegraded":"2023-10-30T12:39:38.071121+0000","last_fullsized":"2023-10-30T12:39:38.071121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:10.773828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":826,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.413791+0000","last_change":"2023-10-30T12:19:40.870027+0000","last_active":"2023-10-30T12:39:40.413791+0000","last_peered":"2023-10-30T12:39:40.413791+0000","last_clean":"2023-10-30T12:39:40.413791+0000","last_became_active":"2023-10-30T12:19:40.864175+0000","last_became_peered":"2023-10-30T12:19:40.864175+0000","last_unstale":"2023-10-30T12:39:40.413791+0000","last_undegraded":"2023-10-30T12:39:40.413791+0000","last_fullsized":"2023-10-30T12:39:40.413791+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:47.649334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":841,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.300250+0000","last_change":"2023-10-30T12:18:01.550154+0000","last_active":"2023-10-30T12:39:40.300250+0000","last_peered":"2023-10-30T12:39:40.300250+0000","last_clean":"2023-10-30T12:39:40.300250+0000","last_became_active":"2023-10-30T12:18:01.523269+0000","last_became_peered":"2023-10-30T12:18:01.523269+0000","last_unstale":"2023-10-30T12:39:40.300250+0000","last_undegraded":"2023-10-30T12:39:40.300250+0000","last_fullsized":"2023-10-30T12:39:40.300250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:08.405234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":834,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.298056+0000","last_change":"2023-10-30T12:19:40.449659+0000","last_active":"2023-10-30T12:39:40.298056+0000","last_peered":"2023-10-30T12:39:40.298056+0000","last_clean":"2023-10-30T12:39:40.298056+0000","last_became_active":"2023-10-30T12:19:40.440094+0000","last_became_peered":"2023-10-30T12:19:40.440094+0000","last_unstale":"2023-10-30T12:39:40.298056+0000","last_undegraded":"2023-10-30T12:39:40.298056+0000","last_fullsized":"2023-10-30T12:39:40.298056+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:12.412748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":851,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.048394+0000","last_change":"2023-10-30T12:18:01.509923+0000","last_active":"2023-10-30T12:39:38.048394+0000","last_peered":"2023-10-30T12:39:38.048394+0000","last_clean":"2023-10-30T12:39:38.048394+0000","last_became_active":"2023-10-30T12:18:01.504543+0000","last_became_peered":"2023-10-30T12:18:01.504543+0000","last_unstale":"2023-10-30T12:39:38.048394+0000","last_undegraded":"2023-10-30T12:39:38.048394+0000","last_fullsized":"2023-10-30T12:39:38.048394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:39.886008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":835,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:39.032349+0000","last_change":"2023-10-30T12:19:29.552208+0000","last_active":"2023-10-30T12:39:39.032349+0000","last_peered":"2023-10-30T12:39:39.032349+0000","last_clean":"2023-10-30T12:39:39.032349+0000","last_became_active":"2023-10-30T12:19:29.543275+0000","last_became_peered":"2023-10-30T12:19:29.543275+0000","last_unstale":"2023-10-30T12:39:39.032349+0000","last_undegraded":"2023-10-30T12:39:39.032349+0000","last_fullsized":"2023-10-30T12:39:39.032349+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:51.785658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":854,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.205204+0000","last_change":"2023-10-30T12:18:00.932461+0000","last_active":"2023-10-30T12:39:41.205204+0000","last_peered":"2023-10-30T12:39:41.205204+0000","last_clean":"2023-10-30T12:39:41.205204+0000","last_became_active":"2023-10-30T12:18:00.904525+0000","last_became_peered":"2023-10-30T12:18:00.904525+0000","last_unstale":"2023-10-30T12:39:41.205204+0000","last_undegraded":"2023-10-30T12:39:41.205204+0000","last_fullsized":"2023-10-30T12:39:41.205204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:42.734208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":827,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.858779+0000","last_change":"2023-10-30T12:19:26.068263+0000","last_active":"2023-10-30T12:39:41.858779+0000","last_peered":"2023-10-30T12:39:41.858779+0000","last_clean":"2023-10-30T12:39:41.858779+0000","last_became_active":"2023-10-30T12:19:26.061689+0000","last_became_peered":"2023-10-30T12:19:26.061689+0000","last_unstale":"2023-10-30T12:39:41.858779+0000","last_undegraded":"2023-10-30T12:39:41.858779+0000","last_fullsized":"2023-10-30T12:39:41.858779+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:50.946689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":851,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.014253+0000","last_change":"2023-10-30T12:18:01.030843+0000","last_active":"2023-10-30T12:39:38.014253+0000","last_peered":"2023-10-30T12:39:38.014253+0000","last_clean":"2023-10-30T12:39:38.014253+0000","last_became_active":"2023-10-30T12:18:01.003382+0000","last_became_peered":"2023-10-30T12:18:01.003382+0000","last_unstale":"2023-10-30T12:39:38.014253+0000","last_undegraded":"2023-10-30T12:39:38.014253+0000","last_fullsized":"2023-10-30T12:39:38.014253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:20.711334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":826,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.521943+0000","last_change":"2023-10-30T12:19:25.299105+0000","last_active":"2023-10-30T12:39:40.521943+0000","last_peered":"2023-10-30T12:39:40.521943+0000","last_clean":"2023-10-30T12:39:40.521943+0000","last_became_active":"2023-10-30T12:19:25.272220+0000","last_became_peered":"2023-10-30T12:19:25.272220+0000","last_unstale":"2023-10-30T12:39:40.521943+0000","last_undegraded":"2023-10-30T12:39:40.521943+0000","last_fullsized":"2023-10-30T12:39:40.521943+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:17.556977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":851,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.056001+0000","last_change":"2023-10-30T12:18:01.406267+0000","last_active":"2023-10-30T12:39:38.056001+0000","last_peered":"2023-10-30T12:39:38.056001+0000","last_clean":"2023-10-30T12:39:38.056001+0000","last_became_active":"2023-10-30T12:18:01.393697+0000","last_became_peered":"2023-10-30T12:18:01.393697+0000","last_unstale":"2023-10-30T12:39:38.056001+0000","last_undegraded":"2023-10-30T12:39:38.056001+0000","last_fullsized":"2023-10-30T12:39:38.056001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:15:29.710122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":941,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.832524+0000","last_change":"2023-10-30T12:17:56.604947+0000","last_active":"2023-10-30T12:39:41.832524+0000","last_peered":"2023-10-30T12:39:41.832524+0000","last_clean":"2023-10-30T12:39:41.832524+0000","last_became_active":"2023-10-30T12:17:56.575838+0000","last_became_peered":"2023-10-30T12:17:56.575838+0000","last_unstale":"2023-10-30T12:39:41.832524+0000","last_undegraded":"2023-10-30T12:39:41.832524+0000","last_fullsized":"2023-10-30T12:39:41.832524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_clean_scrub_stamp":"2023-10-30T12:17:53.592563+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:47.623228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":845,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.080335+0000","last_change":"2023-10-30T12:19:25.736207+0000","last_active":"2023-10-30T12:39:38.080335+0000","last_peered":"2023-10-30T12:39:38.080335+0000","last_clean":"2023-10-30T12:39:38.080335+0000","last_became_active":"2023-10-30T12:19:25.715885+0000","last_became_peered":"2023-10-30T12:19:25.715885+0000","last_unstale":"2023-10-30T12:39:38.080335+0000","last_undegraded":"2023-10-30T12:39:38.080335+0000","last_fullsized":"2023-10-30T12:39:38.080335+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:50.517927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":838,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:39.037645+0000","last_change":"2023-10-30T12:19:29.928614+0000","last_active":"2023-10-30T12:39:39.037645+0000","last_peered":"2023-10-30T12:39:39.037645+0000","last_clean":"2023-10-30T12:39:39.037645+0000","last_became_active":"2023-10-30T12:19:29.879226+0000","last_became_peered":"2023-10-30T12:19:29.879226+0000","last_unstale":"2023-10-30T12:39:39.037645+0000","last_undegraded":"2023-10-30T12:39:39.037645+0000","last_fullsized":"2023-10-30T12:39:39.037645+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:26:19.751528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":843,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.068307+0000","last_change":"2023-10-30T12:19:40.799718+0000","last_active":"2023-10-30T12:39:38.068307+0000","last_peered":"2023-10-30T12:39:38.068307+0000","last_clean":"2023-10-30T12:39:38.068307+0000","last_became_active":"2023-10-30T12:19:40.794341+0000","last_became_peered":"2023-10-30T12:19:40.794341+0000","last_unstale":"2023-10-30T12:39:38.068307+0000","last_undegraded":"2023-10-30T12:39:38.068307+0000","last_fullsized":"2023-10-30T12:39:38.068307+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:58.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":844,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.077563+0000","last_change":"2023-10-30T12:19:28.247323+0000","last_active":"2023-10-30T12:39:38.077563+0000","last_peered":"2023-10-30T12:39:38.077563+0000","last_clean":"2023-10-30T12:39:38.077563+0000","last_became_active":"2023-10-30T12:19:28.238513+0000","last_became_peered":"2023-10-30T12:19:28.238513+0000","last_unstale":"2023-10-30T12:39:38.077563+0000","last_undegraded":"2023-10-30T12:39:38.077563+0000","last_fullsized":"2023-10-30T12:39:38.077563+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:43.483993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":833,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.958239+0000","last_change":"2023-10-30T12:19:27.303856+0000","last_active":"2023-10-30T12:39:38.958239+0000","last_peered":"2023-10-30T12:39:38.958239+0000","last_clean":"2023-10-30T12:39:38.958239+0000","last_became_active":"2023-10-30T12:19:27.284772+0000","last_became_peered":"2023-10-30T12:19:27.284772+0000","last_unstale":"2023-10-30T12:39:38.958239+0000","last_undegraded":"2023-10-30T12:39:38.958239+0000","last_fullsized":"2023-10-30T12:39:38.958239+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:40.340431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":827,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.834340+0000","last_change":"2023-10-30T12:19:33.492307+0000","last_active":"2023-10-30T12:39:41.834340+0000","last_peered":"2023-10-30T12:39:41.834340+0000","last_clean":"2023-10-30T12:39:41.834340+0000","last_became_active":"2023-10-30T12:19:33.483122+0000","last_became_peered":"2023-10-30T12:19:33.483122+0000","last_unstale":"2023-10-30T12:39:41.834340+0000","last_undegraded":"2023-10-30T12:39:41.834340+0000","last_fullsized":"2023-10-30T12:39:41.834340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:16.210880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":839,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.998483+0000","last_change":"2023-10-30T12:19:26.678316+0000","last_active":"2023-10-30T12:39:38.998483+0000","last_peered":"2023-10-30T12:39:38.998483+0000","last_clean":"2023-10-30T12:39:38.998483+0000","last_became_active":"2023-10-30T12:19:26.663005+0000","last_became_peered":"2023-10-30T12:19:26.663005+0000","last_unstale":"2023-10-30T12:39:38.998483+0000","last_undegraded":"2023-10-30T12:39:38.998483+0000","last_fullsized":"2023-10-30T12:39:38.998483+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:22:45.623416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":844,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.011668+0000","last_change":"2023-10-30T12:19:26.773408+0000","last_active":"2023-10-30T12:39:38.011668+0000","last_peered":"2023-10-30T12:39:38.011668+0000","last_clean":"2023-10-30T12:39:38.011668+0000","last_became_active":"2023-10-30T12:19:26.766234+0000","last_became_peered":"2023-10-30T12:19:26.766234+0000","last_unstale":"2023-10-30T12:39:38.011668+0000","last_undegraded":"2023-10-30T12:39:38.011668+0000","last_fullsized":"2023-10-30T12:39:38.011668+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:39.619411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":830,"reported_epoch":892,"state":"active+clean","last_fresh":"2023-10-30T12:39:36.659758+0000","last_change":"2023-10-30T12:19:29.219323+0000","last_active":"2023-10-30T12:39:36.659758+0000","last_peered":"2023-10-30T12:39:36.659758+0000","last_clean":"2023-10-30T12:39:36.659758+0000","last_became_active":"2023-10-30T12:19:29.191499+0000","last_became_peered":"2023-10-30T12:19:29.191499+0000","last_unstale":"2023-10-30T12:39:36.659758+0000","last_undegraded":"2023-10-30T12:39:36.659758+0000","last_fullsized":"2023-10-30T12:39:36.659758+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.259618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":848,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.270235+0000","last_change":"2023-10-30T12:19:30.420980+0000","last_active":"2023-10-30T12:39:41.270235+0000","last_peered":"2023-10-30T12:39:41.270235+0000","last_clean":"2023-10-30T12:39:41.270235+0000","last_became_active":"2023-10-30T12:19:30.408141+0000","last_became_peered":"2023-10-30T12:19:30.408141+0000","last_unstale":"2023-10-30T12:39:41.270235+0000","last_undegraded":"2023-10-30T12:39:41.270235+0000","last_fullsized":"2023-10-30T12:39:41.270235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:04:46.816451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":834,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.223908+0000","last_change":"2023-10-30T12:19:40.539149+0000","last_active":"2023-10-30T12:39:40.223908+0000","last_peered":"2023-10-30T12:39:40.223908+0000","last_clean":"2023-10-30T12:39:40.223908+0000","last_became_active":"2023-10-30T12:19:40.528642+0000","last_became_peered":"2023-10-30T12:19:40.528642+0000","last_unstale":"2023-10-30T12:39:40.223908+0000","last_undegraded":"2023-10-30T12:39:40.223908+0000","last_fullsized":"2023-10-30T12:39:40.223908+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:34.653320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":835,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.295847+0000","last_change":"2023-10-30T12:19:30.748646+0000","last_active":"2023-10-30T12:39:40.295847+0000","last_peered":"2023-10-30T12:39:40.295847+0000","last_clean":"2023-10-30T12:39:40.295847+0000","last_became_active":"2023-10-30T12:19:30.720900+0000","last_became_peered":"2023-10-30T12:19:30.720900+0000","last_unstale":"2023-10-30T12:39:40.295847+0000","last_undegraded":"2023-10-30T12:39:40.295847+0000","last_fullsized":"2023-10-30T12:39:40.295847+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:59.574207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509705,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63332,"kb_used_data":9304,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308508,"statfs":{"total":96636764160,"available":96571912192,"internally_reserved":0,"allocated":9527296,"data_stored":4823141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":23.231999999999999,"5min":17.655999999999999,"15min":16.648},"min":{"1min":6.4530000000000003,"5min":4.0250000000000004,"15min":4.0250000000000004},"max":{"1min":181.905,"5min":233.52500000000001,"15min":233.52500000000001},"last":30.998999999999999},{"interface":"front","average":{"1min":27.143000000000001,"5min":21.59,"15min":22.366},"min":{"1min":9.7650000000000006,"5min":5.6740000000000004,"15min":5.6740000000000004},"max":{"1min":183.072,"5min":214.31899999999999,"15min":365.47399999999999},"last":35.723999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.035,"5min":12.183,"15min":14.696},"min":{"1min":6.0800000000000001,"5min":4.4359999999999999,"15min":4.2389999999999999},"max":{"1min":63.286999999999999,"5min":63.286999999999999,"15min":142.51400000000001},"last":6.6500000000000004},{"interface":"front","average":{"1min":17.978999999999999,"5min":14.823,"15min":17.242000000000001},"min":{"1min":6.056,"5min":5.1689999999999996,"15min":5.1669999999999998},"max":{"1min":61.552,"5min":112.276,"15min":211.11600000000001},"last":7.8440000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.715,"5min":21.622,"15min":20.561},"min":{"1min":6.5259999999999998,"5min":6.2530000000000001,"15min":6.2530000000000001},"max":{"1min":61.927,"5min":187.00399999999999,"15min":187.00399999999999},"last":9.4510000000000005},{"interface":"front","average":{"1min":20.795999999999999,"5min":23.681999999999999,"15min":21.565000000000001},"min":{"1min":8.2710000000000008,"5min":6.5359999999999996,"15min":5.6260000000000003},"max":{"1min":80.686999999999998,"5min":240.99799999999999,"15min":240.99799999999999},"last":8.8460000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":20.48,"5min":26.420000000000002,"15min":25.207999999999998},"min":{"1min":10.49,"5min":7.5990000000000002,"15min":6.2240000000000002},"max":{"1min":62.517000000000003,"5min":279.94099999999997,"15min":279.94099999999997},"last":12.83},{"interface":"front","average":{"1min":17.039000000000001,"5min":24.077999999999999,"15min":23.585000000000001},"min":{"1min":8.1370000000000005,"5min":6.617,"15min":6.5960000000000001},"max":{"1min":55.351999999999997,"5min":180.74600000000001,"15min":180.74600000000001},"last":36.521999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":13.237,"5min":19.042000000000002,"15min":18.274999999999999},"min":{"1min":6.617,"5min":6.617,"15min":6.1840000000000002},"max":{"1min":59.103000000000002,"5min":184.88,"15min":184.88},"last":31.678000000000001},{"interface":"front","average":{"1min":20.768999999999998,"5min":26.350000000000001,"15min":25.687999999999999},"min":{"1min":11.442,"5min":9.4160000000000004,"15min":7.3540000000000001},"max":{"1min":54.426000000000002,"5min":374.38900000000001,"15min":374.38900000000001},"last":13.537000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":19.882000000000001,"5min":25.530999999999999,"15min":25.718},"min":{"1min":10.507,"5min":8.1210000000000004,"15min":6.2729999999999997},"max":{"1min":43.252000000000002,"5min":218.28800000000001,"15min":218.28800000000001},"last":39.472999999999999},{"interface":"front","average":{"1min":17.363,"5min":25.651,"15min":23.794},"min":{"1min":6.7160000000000002,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":34.281999999999996,"5min":195.74700000000001,"15min":195.74700000000001},"last":32.414999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":21.611999999999998,"5min":23.324000000000002,"15min":27.591999999999999},"min":{"1min":9.8620000000000001,"5min":9.593,"15min":8.3320000000000007},"max":{"1min":69.704999999999998,"5min":178.68100000000001,"15min":224.71000000000001},"last":65.272000000000006},{"interface":"front","average":{"1min":19.506,"5min":24.533000000000001,"15min":25.074000000000002},"min":{"1min":9.7799999999999994,"5min":7.2469999999999999,"15min":7.2469999999999999},"max":{"1min":39.325000000000003,"5min":187.14500000000001,"15min":210.05199999999999},"last":33.552999999999997}]}]},{"osd":5,"up_from":15,"seq":64424509705,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82020,"kb_used_data":10132,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289820,"statfs":{"total":96636764160,"available":96552775680,"internally_reserved":0,"allocated":10375168,"data_stored":5538149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":82,"apply_latency_ms":82,"commit_latency_ns":82000000,"apply_latency_ns":82000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:38:43 2023","interfaces":[{"interface":"back","average":{"1min":22.297000000000001,"5min":15.888999999999999,"15min":28.812000000000001},"min":{"1min":4.4119999999999999,"5min":4.1050000000000004,"15min":4.1050000000000004},"max":{"1min":136.36000000000001,"5min":158.87799999999999,"15min":627.18200000000002},"last":12.56},{"interface":"front","average":{"1min":18.611999999999998,"5min":16.850000000000001,"15min":26.960999999999999},"min":{"1min":6.367,"5min":4.7469999999999999,"15min":4.7469999999999999},"max":{"1min":94.358999999999995,"5min":146.374,"15min":341.17000000000002},"last":7.75}]},{"osd":1,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":18.443999999999999,"5min":16.146000000000001,"15min":28.567},"min":{"1min":5.8040000000000003,"5min":5.5300000000000002,"15min":5.5300000000000002},"max":{"1min":138.964,"5min":138.964,"15min":620.21500000000003},"last":4.9160000000000004},{"interface":"front","average":{"1min":17.212,"5min":17.853000000000002,"15min":27.408000000000001},"min":{"1min":7.1970000000000001,"5min":5.6150000000000002,"15min":5.4109999999999996},"max":{"1min":93.744,"5min":162.25200000000001,"15min":334.62},"last":13.255000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":18.535,"5min":18.928000000000001,"15min":23.741},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":134.77000000000001,"5min":208.28100000000001,"15min":246.37},"last":8.5500000000000007},{"interface":"front","average":{"1min":21.039000000000001,"5min":20.257999999999999,"15min":24.73},"min":{"1min":8.4540000000000006,"5min":6.3049999999999997,"15min":5.2770000000000001},"max":{"1min":141.99799999999999,"5min":141.99799999999999,"15min":348.41000000000003},"last":13.888}]},{"osd":3,"last update":"Mon Oct 30 12:39:26 2023","interfaces":[{"interface":"back","average":{"1min":19.34,"5min":23.516999999999999,"15min":25.564},"min":{"1min":6.8339999999999996,"5min":6.3860000000000001,"15min":6.3860000000000001},"max":{"1min":97.822999999999993,"5min":148.125,"15min":273.10300000000001},"last":16.361999999999998},{"interface":"front","average":{"1min":24.561,"5min":21.800000000000001,"15min":24.858000000000001},"min":{"1min":9.3970000000000002,"5min":6.2889999999999997,"15min":5.8029999999999999},"max":{"1min":94.296000000000006,"5min":154.857,"15min":282.28500000000003},"last":14.513}]},{"osd":4,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":25.477,"5min":23.068000000000001,"15min":33.439},"min":{"1min":11.792999999999999,"5min":9.5109999999999992,"15min":7.0259999999999998},"max":{"1min":144.102,"5min":144.102,"15min":512.01199999999994},"last":15.071999999999999},{"interface":"front","average":{"1min":27.440999999999999,"5min":25.501000000000001,"15min":31.102},"min":{"1min":7.4530000000000003,"5min":7.4530000000000003,"15min":6.3890000000000002},"max":{"1min":113.208,"5min":169.45500000000001,"15min":386.62200000000001},"last":9.8640000000000008}]},{"osd":6,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":28.742000000000001,"5min":28.922000000000001,"15min":31.957000000000001},"min":{"1min":14.225,"5min":11.138,"15min":7.9729999999999999},"max":{"1min":144.726,"5min":242.16800000000001,"15min":588.93899999999996},"last":15.701000000000001},{"interface":"front","average":{"1min":32.106999999999999,"5min":34.963999999999999,"15min":37.119},"min":{"1min":16.084,"5min":12.661,"15min":7.25},"max":{"1min":161.18000000000001,"5min":319.17700000000002,"15min":500.173},"last":17.097999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:39:26 2023","interfaces":[{"interface":"back","average":{"1min":20.66,"5min":27.561,"15min":28.265999999999998},"min":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"max":{"1min":113.35899999999999,"5min":258.61200000000002,"15min":258.61200000000002},"last":10.956},{"interface":"front","average":{"1min":16.838999999999999,"5min":22.385999999999999,"15min":24.34},"min":{"1min":6.109,"5min":6.109,"15min":6.109},"max":{"1min":55.759,"5min":145.15000000000001,"15min":245.71899999999999},"last":11.893000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509705,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63564,"kb_used_data":9512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308276,"statfs":{"total":96636764160,"available":96571674624,"internally_reserved":0,"allocated":9740288,"data_stored":4978430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":8.8330000000000002,"5min":15.872,"15min":22.242999999999999},"min":{"1min":4.7569999999999997,"5min":4.2140000000000004,"15min":4.0620000000000003},"max":{"1min":16.516999999999999,"5min":158.499,"15min":295.56400000000002},"last":8.7710000000000008},{"interface":"front","average":{"1min":8.3000000000000007,"5min":18.66,"15min":23.565999999999999},"min":{"1min":4.4080000000000004,"5min":4.4080000000000004,"15min":4.4080000000000004},"max":{"1min":14.081,"5min":182.96600000000001,"15min":368.01900000000001},"last":7.1189999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:38:45 2023","interfaces":[{"interface":"back","average":{"1min":11.939,"5min":17.212,"15min":23.172000000000001},"min":{"1min":5.9829999999999997,"5min":5.6420000000000003,"15min":4.5389999999999997},"max":{"1min":53.116,"5min":171.24100000000001,"15min":322.97800000000001},"last":12.673},{"interface":"front","average":{"1min":17.463999999999999,"5min":20.486999999999998,"15min":22.018000000000001},"min":{"1min":6.9050000000000002,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":105.38800000000001,"5min":168.833,"15min":245.44900000000001},"last":18.204000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:38:51 2023","interfaces":[{"interface":"back","average":{"1min":18.649999999999999,"5min":26.581,"15min":25.079000000000001},"min":{"1min":7.9740000000000002,"5min":6.0039999999999996,"15min":6.0039999999999996},"max":{"1min":112.88,"5min":316.74799999999999,"15min":316.74799999999999},"last":8.2379999999999995},{"interface":"front","average":{"1min":16.074999999999999,"5min":26.199999999999999,"15min":26.724},"min":{"1min":8.0690000000000008,"5min":5.9489999999999998,"15min":5.9489999999999998},"max":{"1min":53.756,"5min":317.55399999999997,"15min":317.55399999999997},"last":14.481999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":14.321999999999999,"5min":23.728999999999999,"15min":28.404},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":6.7270000000000003},"max":{"1min":21.91,"5min":161.197,"15min":383.87099999999998},"last":10.694000000000001},{"interface":"front","average":{"1min":13.757999999999999,"5min":26.026,"15min":26.616},"min":{"1min":8.5,"5min":8.5,"15min":6.6529999999999996},"max":{"1min":18.109999999999999,"5min":260.94200000000001,"15min":284.61700000000002},"last":16.137}]},{"osd":5,"last update":"Mon Oct 30 12:38:45 2023","interfaces":[{"interface":"back","average":{"1min":23.707999999999998,"5min":27.972000000000001,"15min":30.349},"min":{"1min":7.9429999999999996,"5min":7.9429999999999996,"15min":5.7119999999999997},"max":{"1min":183.55000000000001,"5min":185.47200000000001,"15min":261.80200000000002},"last":19.524999999999999},{"interface":"front","average":{"1min":17.798999999999999,"5min":22.954999999999998,"15min":29.309000000000001},"min":{"1min":9.1099999999999994,"5min":6.3760000000000003,"15min":6.3760000000000003},"max":{"1min":102.428,"5min":174.38999999999999,"15min":371.35700000000003},"last":20.347000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":23.914999999999999,"5min":27.789999999999999,"15min":27.952999999999999},"min":{"1min":10.460000000000001,"5min":9.4939999999999998,"15min":8.0060000000000002},"max":{"1min":95.795000000000002,"5min":215.31800000000001,"15min":312.44999999999999},"last":16.774999999999999},{"interface":"front","average":{"1min":19.725999999999999,"5min":28.399999999999999,"15min":27.664000000000001},"min":{"1min":9.859,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":63.609999999999999,"5min":266.50599999999997,"15min":320.50999999999999},"last":11.413}]},{"osd":7,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":17.331,"5min":26.408000000000001,"15min":28.349},"min":{"1min":9.048,"5min":9.048,"15min":7.9370000000000003},"max":{"1min":55.015000000000001,"5min":193.95400000000001,"15min":319.84199999999998},"last":20.981000000000002},{"interface":"front","average":{"1min":15.836,"5min":26.617000000000001,"15min":27.077000000000002},"min":{"1min":10.515000000000001,"5min":8.9489999999999998,"15min":8.5410000000000004},"max":{"1min":28.527000000000001,"5min":252.232,"15min":315.06900000000002},"last":13.680999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509703,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85592,"kb_used_data":9768,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286248,"statfs":{"total":96636764160,"available":96549117952,"internally_reserved":0,"allocated":10002432,"data_stored":5188384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":82,"apply_latency_ms":82,"commit_latency_ns":82000000,"apply_latency_ns":82000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.731999999999999,"5min":21.190000000000001,"15min":21.754999999999999},"min":{"1min":4.609,"5min":4.2460000000000004,"15min":4.1159999999999997},"max":{"1min":210.398,"5min":303.58600000000001,"15min":303.58600000000001},"last":6.9340000000000002},{"interface":"front","average":{"1min":26.295999999999999,"5min":17.559999999999999,"15min":21.213000000000001},"min":{"1min":6.0049999999999999,"5min":4.8819999999999997,"15min":4.1070000000000002},"max":{"1min":187.82400000000001,"5min":187.82400000000001,"15min":268.68799999999999},"last":13.012}]},{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":42.889000000000003,"5min":25.684999999999999,"15min":23.533999999999999},"min":{"1min":7.7990000000000004,"5min":5.6070000000000002,"15min":4.8940000000000001},"max":{"1min":220.96600000000001,"5min":220.96600000000001,"15min":389.303},"last":12.266999999999999},{"interface":"front","average":{"1min":43.094999999999999,"5min":25.721,"15min":24.445},"min":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":5.0890000000000004},"max":{"1min":222.328,"5min":222.328,"15min":468.13299999999998},"last":6.2850000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":19.759,"5min":21.931000000000001,"15min":22.350999999999999},"min":{"1min":8.0250000000000004,"5min":6.8230000000000004,"15min":5.6150000000000002},"max":{"1min":135.97399999999999,"5min":352.09399999999999,"15min":352.09399999999999},"last":9.6430000000000007},{"interface":"front","average":{"1min":18.385000000000002,"5min":20.425999999999998,"15min":23.646999999999998},"min":{"1min":7.7889999999999997,"5min":6.9850000000000003,"15min":6.9850000000000003},"max":{"1min":94.552999999999997,"5min":184.04599999999999,"15min":287.33699999999999},"last":17.280999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.673000000000002,"5min":25.640000000000001,"15min":28.405000000000001},"min":{"1min":8.6929999999999996,"5min":7.1799999999999997,"15min":6.5010000000000003},"max":{"1min":211.03200000000001,"5min":310.12099999999998,"15min":310.12099999999998},"last":8.2880000000000003},{"interface":"front","average":{"1min":31.611000000000001,"5min":25.715,"15min":25.268000000000001},"min":{"1min":9.2690000000000001,"5min":6.7169999999999996,"15min":6.0919999999999996},"max":{"1min":187.15700000000001,"5min":359.87799999999999,"15min":359.87799999999999},"last":8.9420000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":44.430999999999997,"5min":29.558,"15min":30.042999999999999},"min":{"1min":9.8840000000000003,"5min":9,"15min":6.851},"max":{"1min":274.15300000000002,"5min":360.88099999999997,"15min":564.26700000000005},"last":10.279},{"interface":"front","average":{"1min":50.723999999999997,"5min":28.849,"15min":28.324999999999999},"min":{"1min":10.619,"5min":8.8680000000000003,"15min":8.0109999999999992},"max":{"1min":289.68099999999998,"5min":289.68099999999998,"15min":294.392},"last":13.667999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":30.876999999999999,"5min":29.062999999999999,"15min":30.353999999999999},"min":{"1min":11.016,"5min":9.3369999999999997,"15min":8.8149999999999995},"max":{"1min":197.09200000000001,"5min":374.33999999999997,"15min":374.33999999999997},"last":14.846},{"interface":"front","average":{"1min":33.301000000000002,"5min":31.440000000000001,"15min":31.552},"min":{"1min":11.416,"5min":11.416,"15min":8.9079999999999995},"max":{"1min":177.63499999999999,"5min":352.79500000000002,"15min":352.79500000000002},"last":15.577}]},{"osd":7,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":36.420000000000002,"5min":32.140999999999998,"15min":34.933},"min":{"1min":12.868,"5min":11.401999999999999,"15min":8.5099999999999998},"max":{"1min":203.70099999999999,"5min":382.68299999999999,"15min":382.68299999999999},"last":16.646000000000001},{"interface":"front","average":{"1min":24.998999999999999,"5min":29.645,"15min":32.134},"min":{"1min":11.976000000000001,"5min":9.0850000000000009,"15min":8.7699999999999996},"max":{"1min":86.024000000000001,"5min":361.47199999999998,"15min":562.05700000000002},"last":18.204999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509703,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67412,"kb_used_data":9868,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94304428,"statfs":{"total":96636764160,"available":96567734272,"internally_reserved":0,"allocated":10104832,"data_stored":5216115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:02 2023","interfaces":[{"interface":"back","average":{"1min":9.0120000000000005,"5min":15.948,"15min":19.835000000000001},"min":{"1min":4.4260000000000002,"5min":4.4260000000000002,"15min":3.8079999999999998},"max":{"1min":20.417000000000002,"5min":121.476,"15min":337.86700000000002},"last":5.9729999999999999},{"interface":"front","average":{"1min":10.103,"5min":14.271000000000001,"15min":18.513000000000002},"min":{"1min":5.6310000000000002,"5min":4.4279999999999999,"15min":4.1399999999999997},"max":{"1min":18.503,"5min":119.87,"15min":343.983},"last":11.227}]},{"osd":1,"last update":"Mon Oct 30 12:38:53 2023","interfaces":[{"interface":"back","average":{"1min":10.884,"5min":16.09,"15min":23.937000000000001},"min":{"1min":7.0860000000000003,"5min":4.9790000000000001,"15min":4.6559999999999997},"max":{"1min":16.172000000000001,"5min":153.751,"15min":501.19900000000001},"last":12.455},{"interface":"front","average":{"1min":13.513999999999999,"5min":18.212,"15min":22.693999999999999},"min":{"1min":6.9859999999999998,"5min":6.9859999999999998,"15min":6.649},"max":{"1min":34.274000000000001,"5min":122.767,"15min":391.58300000000003},"last":6.9180000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:38:57 2023","interfaces":[{"interface":"back","average":{"1min":13.710000000000001,"5min":21.341999999999999,"15min":27.303000000000001},"min":{"1min":7.7140000000000004,"5min":6.923,"15min":6.6379999999999999},"max":{"1min":34.911999999999999,"5min":173.56899999999999,"15min":414.69299999999998},"last":8.9190000000000005},{"interface":"front","average":{"1min":14.257,"5min":21.814,"15min":28.373000000000001},"min":{"1min":7.484,"5min":7.1769999999999996,"15min":6.6269999999999998},"max":{"1min":32.726999999999997,"5min":197.904,"15min":502.19299999999998},"last":13.968}]},{"osd":4,"last update":"Mon Oct 30 12:38:41 2023","interfaces":[{"interface":"back","average":{"1min":17.321000000000002,"5min":22.783999999999999,"15min":27.199000000000002},"min":{"1min":12.071999999999999,"5min":8.0540000000000003,"15min":6.415},"max":{"1min":36.734999999999999,"5min":157.77199999999999,"15min":331.834},"last":8},{"interface":"front","average":{"1min":10.712999999999999,"5min":16.815999999999999,"15min":20.263999999999999},"min":{"1min":6.5099999999999998,"5min":5.984,"15min":5.984},"max":{"1min":19.161000000000001,"5min":134.452,"15min":179.89099999999999},"last":10.460000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:38:42 2023","interfaces":[{"interface":"back","average":{"1min":16.175000000000001,"5min":25.510999999999999,"15min":34.662999999999997},"min":{"1min":8.6720000000000006,"5min":8.0619999999999994,"15min":7.4500000000000002},"max":{"1min":36.591000000000001,"5min":245.22800000000001,"15min":695.71100000000001},"last":17.966000000000001},{"interface":"front","average":{"1min":18.864999999999998,"5min":27.582000000000001,"15min":36.226999999999997},"min":{"1min":12.657,"5min":9.6349999999999998,"15min":9.3930000000000007},"max":{"1min":37.334000000000003,"5min":241.41999999999999,"15min":578.82500000000005},"last":9.7189999999999994}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":16.959,"5min":19.983000000000001,"15min":20.140000000000001},"min":{"1min":8.1110000000000007,"5min":6.3239999999999998,"15min":6.1100000000000003},"max":{"1min":49.598999999999997,"5min":146.75,"15min":182.97499999999999},"last":11.861000000000001},{"interface":"front","average":{"1min":20.119,"5min":23.864000000000001,"15min":25.939},"min":{"1min":11.311,"5min":9.8819999999999997,"15min":8.7729999999999997},"max":{"1min":45.713000000000001,"5min":154.619,"15min":315.71100000000001},"last":15.045}]},{"osd":7,"last update":"Mon Oct 30 12:38:53 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":18.695,"15min":22.042999999999999},"min":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"max":{"1min":27.41,"5min":123.926,"15min":228.67400000000001},"last":18.713000000000001},{"interface":"front","average":{"1min":19.202000000000002,"5min":25.521000000000001,"15min":29.041},"min":{"1min":11.582000000000001,"5min":10.462999999999999,"15min":9.0589999999999993},"max":{"1min":38.158000000000001,"5min":160.93600000000001,"15min":328.51799999999997},"last":19.545000000000002}]}]},{"osd":1,"up_from":15,"seq":64424509704,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63800,"kb_used_data":10184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308040,"statfs":{"total":96636764160,"available":96571432960,"internally_reserved":0,"allocated":10428416,"data_stored":5595488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":10.207000000000001,"5min":11.294,"15min":18.631},"min":{"1min":5.5519999999999996,"5min":4.3760000000000003,"15min":4.3760000000000003},"max":{"1min":24.184999999999999,"5min":75.006,"15min":294.67099999999999},"last":14.157},{"interface":"front","average":{"1min":11.981999999999999,"5min":11.672000000000001,"15min":17.254000000000001},"min":{"1min":6.4210000000000003,"5min":5.2400000000000002,"15min":4.952},"max":{"1min":27.530999999999999,"5min":61.932000000000002,"15min":241.066},"last":7.1820000000000004}]},{"osd":2,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":16.41,"5min":14.955,"15min":19.184000000000001},"min":{"1min":6.4299999999999997,"5min":6.4299999999999997,"15min":6.0519999999999996},"max":{"1min":67.140000000000001,"5min":75.491,"15min":295.50599999999997},"last":8.3179999999999996},{"interface":"front","average":{"1min":18.949999999999999,"5min":13.51,"15min":19.122},"min":{"1min":6.8810000000000002,"5min":6.1580000000000004,"15min":5.327},"max":{"1min":135.00800000000001,"5min":135.00800000000001,"15min":269.99200000000002},"last":11.167999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":19.390000000000001,"15min":21.367999999999999},"min":{"1min":6.9889999999999999,"5min":6.4779999999999998,"15min":6.4779999999999998},"max":{"1min":25.774999999999999,"5min":310.40800000000002,"15min":551.58000000000004},"last":9.5630000000000006},{"interface":"front","average":{"1min":17.298999999999999,"5min":28.062000000000001,"15min":27.957000000000001},"min":{"1min":9.8819999999999997,"5min":9.8819999999999997,"15min":9.1999999999999993},"max":{"1min":29.579000000000001,"5min":338.88600000000002,"15min":562.06899999999996},"last":15.07}]},{"osd":4,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":14.186,"5min":16.617000000000001,"15min":21.454000000000001},"min":{"1min":7.0469999999999997,"5min":6.7480000000000002,"15min":6.6109999999999998},"max":{"1min":40.465000000000003,"5min":84.212000000000003,"15min":285.26499999999999},"last":20.274000000000001},{"interface":"front","average":{"1min":14.521000000000001,"5min":16.108000000000001,"15min":22.126000000000001},"min":{"1min":9.0500000000000007,"5min":6.7450000000000001,"15min":6.7450000000000001},"max":{"1min":42.087000000000003,"5min":62.616,"15min":327.59699999999998},"last":16.891999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":35.115000000000002,"5min":25.553999999999998,"15min":26.443999999999999},"min":{"1min":9.9410000000000007,"5min":8.1639999999999997,"15min":7.3070000000000004},"max":{"1min":322.59899999999999,"5min":322.59899999999999,"15min":322.59899999999999},"last":16.023},{"interface":"front","average":{"1min":32.095999999999997,"5min":21.616,"15min":24.370999999999999},"min":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"max":{"1min":283.06,"5min":283.06,"15min":286.43799999999999},"last":12.630000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":13.723000000000001,"5min":17.039000000000001,"15min":22.716000000000001},"min":{"1min":7.6059999999999999,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":28.151,"5min":71.662999999999997,"15min":348.91000000000003},"last":10.596},{"interface":"front","average":{"1min":18.815999999999999,"5min":23.582999999999998,"15min":26.449000000000002},"min":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":8.4819999999999993},"max":{"1min":33.466000000000001,"5min":99.231999999999999,"15min":301.99799999999999},"last":19.335000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":21.850999999999999,"5min":21.949000000000002,"15min":25.800999999999998},"min":{"1min":13.108000000000001,"5min":9.2780000000000005,"15min":7.8739999999999997},"max":{"1min":68.974999999999994,"5min":107.703,"15min":305.19900000000001},"last":17.609000000000002},{"interface":"front","average":{"1min":20.207000000000001,"5min":20.809000000000001,"15min":26.821000000000002},"min":{"1min":8.8640000000000008,"5min":8.8640000000000008,"15min":8.0920000000000005},"max":{"1min":70.406999999999996,"5min":106.678,"15min":277.05200000000002},"last":11.855}]}]},{"osd":0,"up_from":15,"seq":64424509704,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63140,"kb_used_data":9544,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308700,"statfs":{"total":96636764160,"available":96572108800,"internally_reserved":0,"allocated":9773056,"data_stored":5007107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":14.013,"5min":16.620999999999999,"15min":19.768000000000001},"min":{"1min":6.117,"5min":4.1799999999999997,"15min":4.1799999999999997},"max":{"1min":73.113,"5min":232.798,"15min":287.096},"last":47.450000000000003},{"interface":"front","average":{"1min":11.015000000000001,"5min":14.513,"15min":22.515999999999998},"min":{"1min":7.6879999999999997,"5min":5.2370000000000001,"15min":4.7590000000000003},"max":{"1min":29.097000000000001,"5min":218.61099999999999,"15min":425.762},"last":44.664999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:39:40 2023","interfaces":[{"interface":"back","average":{"1min":21.314,"5min":17.907,"15min":20.555},"min":{"1min":8.3599999999999994,"5min":5.2480000000000002,"15min":5.2480000000000002},"max":{"1min":78.947999999999993,"5min":228.90799999999999,"15min":299.46600000000001},"last":78.947999999999993},{"interface":"front","average":{"1min":28.640000000000001,"5min":18.164000000000001,"15min":20.786000000000001},"min":{"1min":6.7779999999999996,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":113.76000000000001,"5min":138.13499999999999,"15min":258.45400000000001},"last":87.332999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.376999999999999,"5min":16.640999999999998,"15min":20.597999999999999},"min":{"1min":9.6329999999999991,"5min":6.5499999999999998,"15min":5.2110000000000003},"max":{"1min":41.456000000000003,"5min":166.21600000000001,"15min":264.97199999999998},"last":55.999000000000002},{"interface":"front","average":{"1min":19.736000000000001,"5min":21.084,"15min":25.367000000000001},"min":{"1min":8.4079999999999995,"5min":8.4079999999999995,"15min":7.218},"max":{"1min":54.859000000000002,"5min":230.73699999999999,"15min":258.93200000000002},"last":58.859000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6620000000000008,"5min":18.376000000000001,"15min":22.655999999999999},"min":{"1min":5.681,"5min":5.681,"15min":5.681},"max":{"1min":24.405999999999999,"5min":209.64599999999999,"15min":209.64599999999999},"last":61.680999999999997},{"interface":"front","average":{"1min":20.053999999999998,"5min":25.309000000000001,"15min":23.466999999999999},"min":{"1min":8.8819999999999997,"5min":8.0990000000000002,"15min":5.8499999999999996},"max":{"1min":90.787999999999997,"5min":208.92400000000001,"15min":208.92400000000001},"last":52.835999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.98,"5min":29.106000000000002,"15min":29.109999999999999},"min":{"1min":10.965,"5min":8.9440000000000008,"15min":7.5510000000000002},"max":{"1min":31.939,"5min":245.16300000000001,"15min":285.57100000000003},"last":60.929000000000002},{"interface":"front","average":{"1min":17.673999999999999,"5min":21.956,"15min":28.129999999999999},"min":{"1min":10.518000000000001,"5min":7.2720000000000002,"15min":7.2720000000000002},"max":{"1min":30.361000000000001,"5min":142.25,"15min":292.07900000000001},"last":54.409999999999997}]},{"osd":6,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":15.827999999999999,"5min":20.135999999999999,"15min":21.259},"min":{"1min":7.7000000000000002,"5min":7.4370000000000003,"15min":7.4370000000000003},"max":{"1min":43.283999999999999,"5min":171.99600000000001,"15min":236.40000000000001},"last":60.557000000000002},{"interface":"front","average":{"1min":18.968,"5min":24.116,"15min":25.454999999999998},"min":{"1min":12.252000000000001,"5min":10.470000000000001,"15min":8.9199999999999999},"max":{"1min":31.385000000000002,"5min":178.59999999999999,"15min":220.68700000000001},"last":65.087000000000003}]},{"osd":7,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":19.300999999999998,"5min":20.318000000000001,"15min":24.036000000000001},"min":{"1min":11.853,"5min":9.2100000000000009,"15min":9.2100000000000009},"max":{"1min":32.664999999999999,"5min":143.49600000000001,"15min":245.87100000000001},"last":55.232999999999997},{"interface":"front","average":{"1min":16.231000000000002,"5min":22.702000000000002,"15min":24.879999999999999},"min":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":32.033000000000001,"5min":181.15100000000001,"15min":260.04000000000002},"last":63.960999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575112,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63560,"kb_used_data":9508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308280,"statfs":{"total":96636764160,"available":96571678720,"internally_reserved":0,"allocated":9736192,"data_stored":4984204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":9.4269999999999996,"5min":10.32,"15min":22.806999999999999},"min":{"1min":4.0960000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"max":{"1min":23.024000000000001,"5min":99.075000000000003,"15min":623.43700000000001},"last":6.4779999999999998},{"interface":"front","average":{"1min":10.904,"5min":11.211,"15min":21.835000000000001},"min":{"1min":4.7279999999999998,"5min":4.202,"15min":4.202},"max":{"1min":24.524999999999999,"5min":51.253999999999998,"15min":563.08900000000006},"last":7.6710000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":11.772,"15min":19.108000000000001},"min":{"1min":6.04,"5min":5.0019999999999998,"15min":5.0019999999999998},"max":{"1min":13.172000000000001,"5min":68.641999999999996,"15min":276.80399999999997},"last":7.2800000000000002},{"interface":"front","average":{"1min":13.029999999999999,"5min":16.312999999999999,"15min":24.184000000000001},"min":{"1min":9.3659999999999997,"5min":7.008,"15min":7.008},"max":{"1min":18.454000000000001,"5min":80.891999999999996,"15min":289.52800000000002},"last":13.73}]},{"osd":2,"last update":"Mon Oct 30 12:38:51 2023","interfaces":[{"interface":"back","average":{"1min":11.6,"5min":18.939,"15min":22.053999999999998},"min":{"1min":6.1749999999999998,"5min":6.1749999999999998,"15min":5.0529999999999999},"max":{"1min":20.262,"5min":329.81999999999999,"15min":329.81999999999999},"last":10.012},{"interface":"front","average":{"1min":13.555999999999999,"5min":17.236000000000001,"15min":22.600000000000001},"min":{"1min":7.5229999999999997,"5min":6.8129999999999997,"15min":6.0350000000000001},"max":{"1min":18.695,"5min":132.25800000000001,"15min":245.63},"last":15.695}]},{"osd":3,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":14.976000000000001,"5min":19.353999999999999,"15min":25.904},"min":{"1min":8.6329999999999991,"5min":6.4950000000000001,"15min":6.4950000000000001},"max":{"1min":33.936,"5min":97.286000000000001,"15min":185.98400000000001},"last":16.648},{"interface":"front","average":{"1min":17.303000000000001,"5min":21.863,"15min":28.611999999999998},"min":{"1min":8.2799999999999994,"5min":7.7000000000000002,"15min":6.8159999999999998},"max":{"1min":61.064,"5min":247.65899999999999,"15min":361.625},"last":12.743}]},{"osd":4,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":13.592000000000001,"5min":20.183,"15min":29.582000000000001},"min":{"1min":8.7720000000000002,"5min":5.8150000000000004,"15min":5.8150000000000004},"max":{"1min":17.824000000000002,"5min":117.967,"15min":838.78700000000003},"last":17.553000000000001},{"interface":"front","average":{"1min":12.276,"5min":16.821000000000002,"15min":27.617000000000001},"min":{"1min":6.9329999999999998,"5min":6.9329999999999998,"15min":6.6280000000000001},"max":{"1min":18.846,"5min":82.543999999999997,"15min":836.30899999999997},"last":10.99}]},{"osd":5,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":20.395,"5min":22.149000000000001,"15min":32.493000000000002},"min":{"1min":9.3989999999999991,"5min":7.1440000000000001,"15min":6.5110000000000001},"max":{"1min":64.566000000000003,"5min":186.54499999999999,"15min":504.84100000000001},"last":11.638},{"interface":"front","average":{"1min":24.806999999999999,"5min":22.687999999999999,"15min":32.274000000000001},"min":{"1min":11.058999999999999,"5min":7.5919999999999996,"15min":6.7249999999999996},"max":{"1min":108.858,"5min":185.61099999999999,"15min":405.85000000000002},"last":8.6980000000000004}]},{"osd":7,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":16.547999999999998,"5min":21.190999999999999,"15min":28.661999999999999},"min":{"1min":10.946,"5min":9.3940000000000001,"15min":7.5579999999999998},"max":{"1min":20.657,"5min":99.480999999999995,"15min":402.315},"last":18.251999999999999},{"interface":"front","average":{"1min":17.120000000000001,"5min":24.227,"15min":27.654},"min":{"1min":13.159000000000001,"5min":8.8019999999999996,"15min":8.8019999999999996},"max":{"1min":21.486999999999998,"5min":264.99299999999999,"15min":291.69},"last":14.337999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:39:45.093 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T12:39:45.094 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:39:45.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.192+0000 7f5c01421640 10 monclient: get_monmap_and_config 2023-10-30T12:39:45.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.192+0000 7f5c01421640 10 monclient: build_initial_monmap 2023-10-30T12:39:45.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.192+0000 7f5c01421640 10 monclient: monmap: 2023-10-30T12:39:45.194 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:45.194 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:45.195 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:45.193171+0000 2023-10-30T12:39:45.195 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:45.193171+0000 2023-10-30T12:39:45.195 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:45.195 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:45.195 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:45.196 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:45.196 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:45.196 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:45.196 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:45.196 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.192+0000 7f5c01421640 1 Processor -- start 2023-10-30T12:39:45.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.192+0000 7f5c01421640 1 -- start start 2023-10-30T12:39:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.192+0000 7f5c01421640 10 monclient: init 2023-10-30T12:39:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T12:39:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc0b7c10 0x7f5bfc0a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 10 monclient(hunting): picked mon.noname-b con 0x7f5bfc0b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bfc141940 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:45.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bfc13e730 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:45.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc0a9350 con 0x7f5bfc141940 2023-10-30T12:39:45.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc0aa6a0 con 0x7f5bfc13e730 2023-10-30T12:39:45.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:45.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0670a0 con 0x7f5bfc0b7c10 2023-10-30T12:39:45.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:45.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5c01421640 10 monclient(hunting): authenticate will time out at 2877.398212s 2023-10-30T12:39:45.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5bfa7fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5bfc141940 0x7f5bfc141d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42442/0 (socket says 172.21.15.148:42442) 2023-10-30T12:39:45.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5bfa7fc640 1 -- 172.21.15.148:0/1919985910 learned_addr learned my addr 172.21.15.148:0/1919985910 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:39:45.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.193+0000 7f5bfaffd640 1 --2- 172.21.15.148:0/1919985910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc0b7c10 0x7f5bfc0a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:45.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.194+0000 7f5bfaffd640 10 monclient(hunting): get_auth_request con 0x7f5bfc0b7c10 auth_method 0 2023-10-30T12:39:45.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.194+0000 7f5bfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:45.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.194+0000 7f5bfaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:45.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.194+0000 7f5bfaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:45.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.194+0000 7f5bfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:45.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.194+0000 7f5bfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:45.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.194+0000 7f5bfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:45.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.196+0000 7f5bfaffd640 10 monclient(hunting): handle_auth_done global_id 17687 payload 274 2023-10-30T12:39:45.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.196+0000 7f5bf9ffb640 1 -- 172.21.15.148:0/1919985910 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829524414 0 0) 0x7f5bec001490 con 0x7f5bfc13e730 2023-10-30T12:39:45.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.196+0000 7f5bfaffd640 1 -- 172.21.15.148:0/1919985910 >> v1:172.21.15.150:6789/0 conn(0x7f5bfc13e730 legacy=0x7f5bfc13eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.196+0000 7f5bfaffd640 1 -- 172.21.15.148:0/1919985910 >> v1:172.21.15.148:6789/0 conn(0x7f5bfc141940 legacy=0x7f5bfc141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.196+0000 7f5bfaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:39:45.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.196+0000 7f5bfaffd640 1 monclient: found mon.noname-b 2023-10-30T12:39:45.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.196+0000 7f5bfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:45.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.196+0000 7f5bfaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:45.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.196+0000 7f5bfaffd640 1 -- 172.21.15.148:0/1919985910 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0aa910 con 0x7f5bfc0b7c10 2023-10-30T12:39:45.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.196+0000 7f5bf9ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:39:45.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.198+0000 7f5bfaffd640 1 --2- 172.21.15.148:0/1919985910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc0b7c10 0x7f5bfc0a8aa0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40097d0 tx=0x7f5bf402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d710ad45b3805554 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:45.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 1 -- 172.21.15.148:0/1919985910 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4031800 con 0x7f5bfc0b7c10 2023-10-30T12:39:45.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:45.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:45.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 10 monclient: dump: 2023-10-30T12:39:45.210 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:45.210 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:45.210 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.210 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.210 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:45.211 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:45.211 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:45.211 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:45.211 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:45.211 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:39:45.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:39:45.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:45.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 1 -- 172.21.15.148:0/1919985910 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf4031970 con 0x7f5bfc0b7c10 2023-10-30T12:39:45.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5c01421640 5 monclient: authenticate success, global_id 17687 2023-10-30T12:39:45.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5c01421640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:39:45.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:45.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5c01421640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:45.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 1 -- 172.21.15.148:0/1919985910 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4031d10 con 0x7f5bfc0b7c10 2023-10-30T12:39:45.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:45.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5c01421640 4 set_mon_vals no callback set 2023-10-30T12:39:45.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:45.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.201+0000 7f5bf9ffb640 10 monclient: dump: 2023-10-30T12:39:45.214 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:45.215 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:45.215 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.215 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.215 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:45.215 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:45.216 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:45.216 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:45.216 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:45.216 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.202+0000 7f5c01421640 10 monclient: shutdown 2023-10-30T12:39:45.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.202+0000 7f5c01421640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:45.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.202+0000 7f5c01421640 1 -- 172.21.15.148:0/1919985910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc0b7c10 msgr2=0x7f5bfc0a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.202+0000 7f5c01421640 1 --2- 172.21.15.148:0/1919985910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc0b7c10 0x7f5bfc0a8aa0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40097d0 tx=0x7f5bf402b8a0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.202+0000 7f5c01421640 1 -- 172.21.15.148:0/1919985910 shutdown_connections 2023-10-30T12:39:45.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.202+0000 7f5c01421640 1 --2- 172.21.15.148:0/1919985910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc0b7c10 0x7f5bfc0a8aa0 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.202+0000 7f5c01421640 1 -- 172.21.15.148:0/1919985910 >> 172.21.15.148:0/1919985910 conn(0x7f5bfc06c660 msgr2=0x7f5bfc0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:45.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.202+0000 7f5c01421640 1 -- 172.21.15.148:0/1919985910 shutdown_connections 2023-10-30T12:39:45.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.202+0000 7f5c01421640 1 -- 172.21.15.148:0/1919985910 wait complete. 2023-10-30T12:39:45.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.203+0000 7f5c01421640 10 monclient: build_initial_monmap 2023-10-30T12:39:45.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.203+0000 7f5c01421640 10 monclient: monmap: 2023-10-30T12:39:45.219 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:45.219 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:45.219 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:45.219 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:45.219 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:45.220 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:45.220 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:45.220 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:45.220 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:45.220 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.203+0000 7f5c01421640 1 librados: starting msgr at 2023-10-30T12:39:45.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.203+0000 7f5c01421640 1 librados: starting objecter 2023-10-30T12:39:45.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.203+0000 7f5c01421640 1 Processor -- start 2023-10-30T12:39:45.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 1 -- start start 2023-10-30T12:39:45.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 1 librados: setting wanted keys 2023-10-30T12:39:45.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 1 librados: calling monclient init 2023-10-30T12:39:45.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient: init 2023-10-30T12:39:45.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:45.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:39:45.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bfc0b7c10 0x7f5bfc1cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:45.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient(hunting): picked mon.noname-b con 0x7f5bfc0b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:45.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc13e730 0x7f5bfc1caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:45.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient(hunting): picked mon.noname-c con 0x7f5bfc13e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:45.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bfc141940 0x7f5bfc1cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:45.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient(hunting): picked mon.noname-a con 0x7f5bfc141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:45.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:45.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0b2fd0 con 0x7f5bfc141940 2023-10-30T12:39:45.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:45.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0ab2d0 con 0x7f5bfc0b7c10 2023-10-30T12:39:45.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:45.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1cc410 con 0x7f5bfc13e730 2023-10-30T12:39:45.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:45.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5c01421640 10 monclient(hunting): authenticate will time out at 2877.409450s 2023-10-30T12:39:45.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.204+0000 7f5bfa7fc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc13e730 0x7f5bfc1caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:45.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bfc141940 0x7f5bfc1cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:45.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfa7fc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc13e730 0x7f5bfc1caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:53910/0 (socket says 172.21.15.148:53910) 2023-10-30T12:39:45.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfa7fc640 1 -- 172.21.15.148:0/1817279726 learned_addr learned my addr 172.21.15.148:0/1817279726 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:45.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfa7fc640 10 monclient(hunting): get_auth_request con 0x7f5bfc13e730 auth_method 0 2023-10-30T12:39:45.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:45.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfa7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:45.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:45.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfb7fe640 10 monclient(hunting): get_auth_request con 0x7f5bfc141940 auth_method 0 2023-10-30T12:39:45.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:45.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:45.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:45.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfaffd640 1 --2- 172.21.15.148:0/1817279726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bfc0b7c10 0x7f5bfc1cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:45.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:45.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:45.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:45.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfaffd640 10 monclient(hunting): get_auth_request con 0x7f5bfc0b7c10 auth_method 0 2023-10-30T12:39:45.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:45.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:45.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:45.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:45.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:45.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.205+0000 7f5bfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:45.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.206+0000 7f5bfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:45.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.206+0000 7f5bfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:45.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.206+0000 7f5bfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:45.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.208+0000 7f5bfa7fc640 10 monclient(hunting): handle_auth_done global_id 17690 payload 824 2023-10-30T12:39:45.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.208+0000 7f5bfa7fc640 1 -- 172.21.15.148:0/1817279726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bfc0b7c10 msgr2=0x7f5bfc1cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.208+0000 7f5bfa7fc640 1 --2- 172.21.15.148:0/1817279726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bfc0b7c10 0x7f5bfc1cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.208+0000 7f5bfa7fc640 1 -- 172.21.15.148:0/1817279726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bfc141940 msgr2=0x7f5bfc1cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.208+0000 7f5bfa7fc640 1 --2- 172.21.15.148:0/1817279726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bfc141940 0x7f5bfc1cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.208+0000 7f5bfa7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:39:45.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.208+0000 7f5bfa7fc640 1 monclient: found mon.noname-c 2023-10-30T12:39:45.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.208+0000 7f5bfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:45.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.208+0000 7f5bfa7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:39:45.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.208+0000 7f5bfa7fc640 1 -- 172.21.15.148:0/1817279726 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0ab580 con 0x7f5bfc13e730 2023-10-30T12:39:45.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.210+0000 7f5bfa7fc640 1 --2- 172.21.15.148:0/1817279726 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc13e730 0x7f5bfc1caff0 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000c4b0 tx=0x7f5bf000c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:45.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.210+0000 7f5be37fe640 10 client.?.objecter ms_handle_connect 0x7f5bfc13e730 2023-10-30T12:39:45.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.210+0000 7f5be37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:45.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5be37fe640 1 -- 172.21.15.148:0/1817279726 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0005ce0 con 0x7f5bfc13e730 2023-10-30T12:39:45.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5be37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:45.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5be37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:45.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5be37fe640 10 monclient: dump: 2023-10-30T12:39:45.234 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:45.234 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:45.234 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.235 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.235 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:45.235 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:45.235 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:45.235 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:45.236 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:45.236 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5be37fe640 10 monclient: _finish_auth 0 2023-10-30T12:39:45.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5be37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:39:45.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5be37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:45.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5c01421640 5 monclient: authenticate success, global_id 17690 2023-10-30T12:39:45.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5c01421640 10 monclient: _renew_subs 2023-10-30T12:39:45.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5c01421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:45.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5c01421640 1 -- 172.21.15.148:0/1817279726 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc1cd800 con 0x7f5bfc13e730 2023-10-30T12:39:45.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5c01421640 10 client.17690.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:45.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5c01421640 10 monclient: _renew_subs 2023-10-30T12:39:45.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5c01421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:45.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5c01421640 1 -- 172.21.15.148:0/1817279726 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc1cbc50 con 0x7f5bfc13e730 2023-10-30T12:39:45.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.212+0000 7f5c01421640 1 librados: init done 2023-10-30T12:39:45.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.213+0000 7f5c01421640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:45.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.213+0000 7f5c01421640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:45.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.213+0000 7f5c01421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:45.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.213+0000 7f5c01421640 1 -- 172.21.15.148:0/1817279726 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc0ac800 con 0x7f5bfc13e730 2023-10-30T12:39:45.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.214+0000 7f5be37fe640 1 -- 172.21.15.148:0/1817279726 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf000e040 con 0x7f5bfc13e730 2023-10-30T12:39:45.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.214+0000 7f5be37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:45.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.214+0000 7f5be37fe640 1 -- 172.21.15.148:0/1817279726 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0007270 con 0x7f5bfc13e730 2023-10-30T12:39:45.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.214+0000 7f5be37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:45.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.214+0000 7f5be37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:45.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.214+0000 7f5be37fe640 10 monclient: dump: 2023-10-30T12:39:45.240 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:45.240 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:45.240 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.241 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.241 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:45.241 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:45.241 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:45.241 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:45.242 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:45.242 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.214+0000 7f5bf97fa640 4 set_mon_vals no callback set 2023-10-30T12:39:45.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.465+0000 7f5be37fe640 1 -- 172.21.15.148:0/1817279726 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f5bf00074a0 con 0x7f5bfc13e730 2023-10-30T12:39:45.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.465+0000 7f5be37fe640 1 --2- 172.21.15.148:0/1817279726 >> v2:172.21.15.148:6800/3205090179 conn(0x7f5be806c7c0 0x7f5be806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:45.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.466+0000 7f5bfaffd640 1 --2- 172.21.15.148:0/1817279726 >> v2:172.21.15.148:6800/3205090179 conn(0x7f5be806c7c0 0x7f5be806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:45.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.466+0000 7f5bfaffd640 10 monclient: get_auth_request con 0x7f5be806c7c0 auth_method 0 2023-10-30T12:39:45.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.466+0000 7f5be37fe640 10 client.17690.objecter ms_handle_connect 0x7f5be806c7c0 2023-10-30T12:39:45.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.466+0000 7f5bfaffd640 1 --2- 172.21.15.148:0/1817279726 >> v2:172.21.15.148:6800/3205090179 conn(0x7f5be806c7c0 0x7f5be806ec80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4036000 tx=0x7f5bf403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.479+0000 7f5be37fe640 1 -- 172.21.15.148:0/1817279726 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(899..899 src has 501..899) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f5bf0057980 con 0x7f5bfc13e730 2023-10-30T12:39:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.479+0000 7f5be37fe640 10 client.17690.objecter ms_dispatch 0x7f5bfc000b90 osd_map(899..899 src has 501..899) v4 2023-10-30T12:39:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.479+0000 7f5be37fe640 3 client.17690.objecter handle_osd_map got epochs [899,899] > 0 2023-10-30T12:39:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.479+0000 7f5be37fe640 3 client.17690.objecter handle_osd_map decoding full epoch 899 2023-10-30T12:39:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.479+0000 7f5be37fe640 20 client.17690.objecter dump_active .. 0 homeless 2023-10-30T12:39:45.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.479+0000 7f5be37fe640 1 -- 172.21.15.148:0/1817279726 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5bf005ba20 con 0x7f5bfc13e730 2023-10-30T12:39:45.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.479+0000 7f5be37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:45.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.479+0000 7f5be37fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.657+0000 7f5c01421640 1 -- 172.21.15.148:0/1817279726 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5bfc13f290 con 0x7f5be806c7c0 2023-10-30T12:39:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.662+0000 7f5be37fe640 1 -- 172.21.15.148:0/1817279726 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151928 (secure 0 0 0) 0x7f5bf403a420 con 0x7f5be806c7c0 2023-10-30T12:39:45.665 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:39:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T12:39:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 10 librados: watch_flush enter 2023-10-30T12:39:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 10 librados: watch_flush exit 2023-10-30T12:39:45.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 20 client.17690.objecter shutdown clearing up homeless session... 2023-10-30T12:39:45.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 10 client.17690.objecter successfully canceled tick 2023-10-30T12:39:45.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 -- 172.21.15.148:0/1817279726 >> v2:172.21.15.148:6800/3205090179 conn(0x7f5be806c7c0 msgr2=0x7f5be806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 --2- 172.21.15.148:0/1817279726 >> v2:172.21.15.148:6800/3205090179 conn(0x7f5be806c7c0 0x7f5be806ec80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4036000 tx=0x7f5bf403c040 comp rx=0 tx=0).stop 2023-10-30T12:39:45.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 10 monclient: shutdown 2023-10-30T12:39:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 -- 172.21.15.148:0/1817279726 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc13e730 msgr2=0x7f5bfc1caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 --2- 172.21.15.148:0/1817279726 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc13e730 0x7f5bfc1caff0 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000c4b0 tx=0x7f5bf000c970 comp rx=0 tx=0).stop 2023-10-30T12:39:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 -- 172.21.15.148:0/1817279726 shutdown_connections 2023-10-30T12:39:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 --2- 172.21.15.148:0/1817279726 >> v2:172.21.15.148:6800/3205090179 conn(0x7f5be806c7c0 0x7f5be806ec80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 --2- 172.21.15.148:0/1817279726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bfc141940 0x7f5bfc1cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 --2- 172.21.15.148:0/1817279726 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5bfc13e730 0x7f5bfc1caff0 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 --2- 172.21.15.148:0/1817279726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bfc0b7c10 0x7f5bfc1cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 -- 172.21.15.148:0/1817279726 >> 172.21.15.148:0/1817279726 conn(0x7f5bfc06c660 msgr2=0x7f5bfc0b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:45.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 -- 172.21.15.148:0/1817279726 shutdown_connections 2023-10-30T12:39:45.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 -- 172.21.15.148:0/1817279726 wait complete. 2023-10-30T12:39:45.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.664+0000 7f5c01421640 1 librados: shutdown 2023-10-30T12:39:45.677 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1561,"stamp":"2023-10-30T12:39:45.310218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":551044,"kb_used_data":76116,"kb_used_omap":12,"kb_used_meta":471411,"kb_avail":754423676,"statfs":{"total":773094113280,"available":772529844224,"internally_reserved":0,"allocated":77942784,"data_stored":40164640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":482725433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":423,"apply_latency_ms":423,"commit_latency_ns":423000000,"apply_latency_ns":423000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.299096"},"pg_stats":[{"pgid":"20.8","version":"0'0","reported_seq":839,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:39:42.433158+0000","last_change":"2023-10-30T12:19:35.856437+0000","last_active":"2023-10-30T12:39:42.433158+0000","last_peered":"2023-10-30T12:39:42.433158+0000","last_clean":"2023-10-30T12:39:42.433158+0000","last_became_active":"2023-10-30T12:19:35.850666+0000","last_became_peered":"2023-10-30T12:19:35.850666+0000","last_unstale":"2023-10-30T12:39:42.433158+0000","last_undegraded":"2023-10-30T12:39:42.433158+0000","last_fullsized":"2023-10-30T12:39:42.433158+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:24.452037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":839,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:39:42.436962+0000","last_change":"2023-10-30T12:19:38.560827+0000","last_active":"2023-10-30T12:39:42.436962+0000","last_peered":"2023-10-30T12:39:42.436962+0000","last_clean":"2023-10-30T12:39:42.436962+0000","last_became_active":"2023-10-30T12:19:38.546595+0000","last_became_peered":"2023-10-30T12:19:38.546595+0000","last_unstale":"2023-10-30T12:39:42.436962+0000","last_undegraded":"2023-10-30T12:39:42.436962+0000","last_fullsized":"2023-10-30T12:39:42.436962+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:22.315389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":839,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:39:42.440149+0000","last_change":"2023-10-30T12:19:35.881177+0000","last_active":"2023-10-30T12:39:42.440149+0000","last_peered":"2023-10-30T12:39:42.440149+0000","last_clean":"2023-10-30T12:39:42.440149+0000","last_became_active":"2023-10-30T12:19:35.871000+0000","last_became_peered":"2023-10-30T12:19:35.871000+0000","last_unstale":"2023-10-30T12:39:42.440149+0000","last_undegraded":"2023-10-30T12:39:42.440149+0000","last_fullsized":"2023-10-30T12:39:42.440149+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:16.108884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":840,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:39.015181+0000","last_change":"2023-10-30T12:19:29.602055+0000","last_active":"2023-10-30T12:39:39.015181+0000","last_peered":"2023-10-30T12:39:39.015181+0000","last_clean":"2023-10-30T12:39:39.015181+0000","last_became_active":"2023-10-30T12:19:29.594505+0000","last_became_peered":"2023-10-30T12:19:29.594505+0000","last_unstale":"2023-10-30T12:39:39.015181+0000","last_undegraded":"2023-10-30T12:39:39.015181+0000","last_fullsized":"2023-10-30T12:39:39.015181+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:15:25.873922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":847,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.256860+0000","last_change":"2023-10-30T12:19:27.523347+0000","last_active":"2023-10-30T12:39:41.256860+0000","last_peered":"2023-10-30T12:39:41.256860+0000","last_clean":"2023-10-30T12:39:41.256860+0000","last_became_active":"2023-10-30T12:19:27.515739+0000","last_became_peered":"2023-10-30T12:19:27.515739+0000","last_unstale":"2023-10-30T12:39:41.256860+0000","last_undegraded":"2023-10-30T12:39:41.256860+0000","last_fullsized":"2023-10-30T12:39:41.256860+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:36.852443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":835,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.922009+0000","last_change":"2023-10-30T12:19:31.914796+0000","last_active":"2023-10-30T12:39:41.922009+0000","last_peered":"2023-10-30T12:39:41.922009+0000","last_clean":"2023-10-30T12:39:41.922009+0000","last_became_active":"2023-10-30T12:19:31.894071+0000","last_became_peered":"2023-10-30T12:19:31.894071+0000","last_unstale":"2023-10-30T12:39:41.922009+0000","last_undegraded":"2023-10-30T12:39:41.922009+0000","last_fullsized":"2023-10-30T12:39:41.922009+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:22.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":836,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:39:42.072670+0000","last_change":"2023-10-30T12:19:28.154596+0000","last_active":"2023-10-30T12:39:42.072670+0000","last_peered":"2023-10-30T12:39:42.072670+0000","last_clean":"2023-10-30T12:39:42.072670+0000","last_became_active":"2023-10-30T12:19:28.142591+0000","last_became_peered":"2023-10-30T12:19:28.142591+0000","last_unstale":"2023-10-30T12:39:42.072670+0000","last_undegraded":"2023-10-30T12:39:42.072670+0000","last_fullsized":"2023-10-30T12:39:42.072670+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:46:05.070599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":836,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.426476+0000","last_change":"2023-10-30T12:19:38.653024+0000","last_active":"2023-10-30T12:39:43.426476+0000","last_peered":"2023-10-30T12:39:43.426476+0000","last_clean":"2023-10-30T12:39:43.426476+0000","last_became_active":"2023-10-30T12:19:38.643834+0000","last_became_peered":"2023-10-30T12:19:38.643834+0000","last_unstale":"2023-10-30T12:39:43.426476+0000","last_undegraded":"2023-10-30T12:39:43.426476+0000","last_fullsized":"2023-10-30T12:39:43.426476+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:54.052947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":841,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.989491+0000","last_change":"2023-10-30T12:19:29.499316+0000","last_active":"2023-10-30T12:39:38.989491+0000","last_peered":"2023-10-30T12:39:38.989491+0000","last_clean":"2023-10-30T12:39:38.989491+0000","last_became_active":"2023-10-30T12:19:29.492043+0000","last_became_peered":"2023-10-30T12:19:29.492043+0000","last_unstale":"2023-10-30T12:39:38.989491+0000","last_undegraded":"2023-10-30T12:39:38.989491+0000","last_fullsized":"2023-10-30T12:39:38.989491+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:49.202008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1","version":"38'7","reported_seq":840,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.453703+0000","last_change":"2023-10-30T12:19:25.218622+0000","last_active":"2023-10-30T12:39:40.453703+0000","last_peered":"2023-10-30T12:39:40.453703+0000","last_clean":"2023-10-30T12:39:40.453703+0000","last_became_active":"2023-10-30T12:19:25.212397+0000","last_became_peered":"2023-10-30T12:19:25.212397+0000","last_unstale":"2023-10-30T12:39:40.453703+0000","last_undegraded":"2023-10-30T12:39:40.453703+0000","last_fullsized":"2023-10-30T12:39:40.453703+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:21.606857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":845,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.046753+0000","last_change":"2023-10-30T12:19:29.500965+0000","last_active":"2023-10-30T12:39:38.046753+0000","last_peered":"2023-10-30T12:39:38.046753+0000","last_clean":"2023-10-30T12:39:38.046753+0000","last_became_active":"2023-10-30T12:19:29.491706+0000","last_became_peered":"2023-10-30T12:19:29.491706+0000","last_unstale":"2023-10-30T12:39:38.046753+0000","last_undegraded":"2023-10-30T12:39:38.046753+0000","last_fullsized":"2023-10-30T12:39:38.046753+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:49.491590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":848,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.236376+0000","last_change":"2023-10-30T12:19:32.994118+0000","last_active":"2023-10-30T12:39:41.236376+0000","last_peered":"2023-10-30T12:39:41.236376+0000","last_clean":"2023-10-30T12:39:41.236376+0000","last_became_active":"2023-10-30T12:19:32.988808+0000","last_became_peered":"2023-10-30T12:19:32.988808+0000","last_unstale":"2023-10-30T12:39:41.236376+0000","last_undegraded":"2023-10-30T12:39:41.236376+0000","last_fullsized":"2023-10-30T12:39:41.236376+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:25.605041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":847,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.234304+0000","last_change":"2023-10-30T12:19:28.922326+0000","last_active":"2023-10-30T12:39:41.234304+0000","last_peered":"2023-10-30T12:39:41.234304+0000","last_clean":"2023-10-30T12:39:41.234304+0000","last_became_active":"2023-10-30T12:19:28.916380+0000","last_became_peered":"2023-10-30T12:19:28.916380+0000","last_unstale":"2023-10-30T12:39:41.234304+0000","last_undegraded":"2023-10-30T12:39:41.234304+0000","last_fullsized":"2023-10-30T12:39:41.234304+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:38:19.847301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":851,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.624076+0000","last_change":"2023-10-30T12:18:01.574170+0000","last_active":"2023-10-30T12:39:40.624076+0000","last_peered":"2023-10-30T12:39:40.624076+0000","last_clean":"2023-10-30T12:39:40.624076+0000","last_became_active":"2023-10-30T12:18:01.568787+0000","last_became_peered":"2023-10-30T12:18:01.568787+0000","last_unstale":"2023-10-30T12:39:40.624076+0000","last_undegraded":"2023-10-30T12:39:40.624076+0000","last_fullsized":"2023-10-30T12:39:40.624076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:12.243846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":836,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:39:42.048079+0000","last_change":"2023-10-30T12:19:40.949818+0000","last_active":"2023-10-30T12:39:42.048079+0000","last_peered":"2023-10-30T12:39:42.048079+0000","last_clean":"2023-10-30T12:39:42.048079+0000","last_became_active":"2023-10-30T12:19:40.940463+0000","last_became_peered":"2023-10-30T12:19:40.940463+0000","last_unstale":"2023-10-30T12:39:42.048079+0000","last_undegraded":"2023-10-30T12:39:42.048079+0000","last_fullsized":"2023-10-30T12:39:42.048079+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:43:05.246911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":842,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:39:42.070772+0000","last_change":"2023-10-30T12:18:01.632325+0000","last_active":"2023-10-30T12:39:42.070772+0000","last_peered":"2023-10-30T12:39:42.070772+0000","last_clean":"2023-10-30T12:39:42.070772+0000","last_became_active":"2023-10-30T12:18:01.602190+0000","last_became_peered":"2023-10-30T12:18:01.602190+0000","last_unstale":"2023-10-30T12:39:42.070772+0000","last_undegraded":"2023-10-30T12:39:42.070772+0000","last_fullsized":"2023-10-30T12:39:42.070772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:29:50.132707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":844,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.051655+0000","last_change":"2023-10-30T12:19:40.722588+0000","last_active":"2023-10-30T12:39:38.051655+0000","last_peered":"2023-10-30T12:39:38.051655+0000","last_clean":"2023-10-30T12:39:38.051655+0000","last_became_active":"2023-10-30T12:19:40.713186+0000","last_became_peered":"2023-10-30T12:19:40.713186+0000","last_unstale":"2023-10-30T12:39:38.051655+0000","last_undegraded":"2023-10-30T12:39:38.051655+0000","last_fullsized":"2023-10-30T12:39:38.051655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:38:31.344804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":850,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.071121+0000","last_change":"2023-10-30T12:18:01.060414+0000","last_active":"2023-10-30T12:39:38.071121+0000","last_peered":"2023-10-30T12:39:38.071121+0000","last_clean":"2023-10-30T12:39:38.071121+0000","last_became_active":"2023-10-30T12:18:01.051697+0000","last_became_peered":"2023-10-30T12:18:01.051697+0000","last_unstale":"2023-10-30T12:39:38.071121+0000","last_undegraded":"2023-10-30T12:39:38.071121+0000","last_fullsized":"2023-10-30T12:39:38.071121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:10.773828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":826,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.413791+0000","last_change":"2023-10-30T12:19:40.870027+0000","last_active":"2023-10-30T12:39:40.413791+0000","last_peered":"2023-10-30T12:39:40.413791+0000","last_clean":"2023-10-30T12:39:40.413791+0000","last_became_active":"2023-10-30T12:19:40.864175+0000","last_became_peered":"2023-10-30T12:19:40.864175+0000","last_unstale":"2023-10-30T12:39:40.413791+0000","last_undegraded":"2023-10-30T12:39:40.413791+0000","last_fullsized":"2023-10-30T12:39:40.413791+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:47.649334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":841,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.300250+0000","last_change":"2023-10-30T12:18:01.550154+0000","last_active":"2023-10-30T12:39:40.300250+0000","last_peered":"2023-10-30T12:39:40.300250+0000","last_clean":"2023-10-30T12:39:40.300250+0000","last_became_active":"2023-10-30T12:18:01.523269+0000","last_became_peered":"2023-10-30T12:18:01.523269+0000","last_unstale":"2023-10-30T12:39:40.300250+0000","last_undegraded":"2023-10-30T12:39:40.300250+0000","last_fullsized":"2023-10-30T12:39:40.300250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:08.405234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":834,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.298056+0000","last_change":"2023-10-30T12:19:40.449659+0000","last_active":"2023-10-30T12:39:40.298056+0000","last_peered":"2023-10-30T12:39:40.298056+0000","last_clean":"2023-10-30T12:39:40.298056+0000","last_became_active":"2023-10-30T12:19:40.440094+0000","last_became_peered":"2023-10-30T12:19:40.440094+0000","last_unstale":"2023-10-30T12:39:40.298056+0000","last_undegraded":"2023-10-30T12:39:40.298056+0000","last_fullsized":"2023-10-30T12:39:40.298056+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:12.412748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":851,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.048394+0000","last_change":"2023-10-30T12:18:01.509923+0000","last_active":"2023-10-30T12:39:38.048394+0000","last_peered":"2023-10-30T12:39:38.048394+0000","last_clean":"2023-10-30T12:39:38.048394+0000","last_became_active":"2023-10-30T12:18:01.504543+0000","last_became_peered":"2023-10-30T12:18:01.504543+0000","last_unstale":"2023-10-30T12:39:38.048394+0000","last_undegraded":"2023-10-30T12:39:38.048394+0000","last_fullsized":"2023-10-30T12:39:38.048394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:39.886008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":835,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:39.032349+0000","last_change":"2023-10-30T12:19:29.552208+0000","last_active":"2023-10-30T12:39:39.032349+0000","last_peered":"2023-10-30T12:39:39.032349+0000","last_clean":"2023-10-30T12:39:39.032349+0000","last_became_active":"2023-10-30T12:19:29.543275+0000","last_became_peered":"2023-10-30T12:19:29.543275+0000","last_unstale":"2023-10-30T12:39:39.032349+0000","last_undegraded":"2023-10-30T12:39:39.032349+0000","last_fullsized":"2023-10-30T12:39:39.032349+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:51.785658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":854,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.205204+0000","last_change":"2023-10-30T12:18:00.932461+0000","last_active":"2023-10-30T12:39:41.205204+0000","last_peered":"2023-10-30T12:39:41.205204+0000","last_clean":"2023-10-30T12:39:41.205204+0000","last_became_active":"2023-10-30T12:18:00.904525+0000","last_became_peered":"2023-10-30T12:18:00.904525+0000","last_unstale":"2023-10-30T12:39:41.205204+0000","last_undegraded":"2023-10-30T12:39:41.205204+0000","last_fullsized":"2023-10-30T12:39:41.205204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:42.734208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":827,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.858779+0000","last_change":"2023-10-30T12:19:26.068263+0000","last_active":"2023-10-30T12:39:41.858779+0000","last_peered":"2023-10-30T12:39:41.858779+0000","last_clean":"2023-10-30T12:39:41.858779+0000","last_became_active":"2023-10-30T12:19:26.061689+0000","last_became_peered":"2023-10-30T12:19:26.061689+0000","last_unstale":"2023-10-30T12:39:41.858779+0000","last_undegraded":"2023-10-30T12:39:41.858779+0000","last_fullsized":"2023-10-30T12:39:41.858779+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:50.946689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":851,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.014253+0000","last_change":"2023-10-30T12:18:01.030843+0000","last_active":"2023-10-30T12:39:38.014253+0000","last_peered":"2023-10-30T12:39:38.014253+0000","last_clean":"2023-10-30T12:39:38.014253+0000","last_became_active":"2023-10-30T12:18:01.003382+0000","last_became_peered":"2023-10-30T12:18:01.003382+0000","last_unstale":"2023-10-30T12:39:38.014253+0000","last_undegraded":"2023-10-30T12:39:38.014253+0000","last_fullsized":"2023-10-30T12:39:38.014253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:20.711334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":826,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.521943+0000","last_change":"2023-10-30T12:19:25.299105+0000","last_active":"2023-10-30T12:39:40.521943+0000","last_peered":"2023-10-30T12:39:40.521943+0000","last_clean":"2023-10-30T12:39:40.521943+0000","last_became_active":"2023-10-30T12:19:25.272220+0000","last_became_peered":"2023-10-30T12:19:25.272220+0000","last_unstale":"2023-10-30T12:39:40.521943+0000","last_undegraded":"2023-10-30T12:39:40.521943+0000","last_fullsized":"2023-10-30T12:39:40.521943+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:17.556977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":851,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.056001+0000","last_change":"2023-10-30T12:18:01.406267+0000","last_active":"2023-10-30T12:39:38.056001+0000","last_peered":"2023-10-30T12:39:38.056001+0000","last_clean":"2023-10-30T12:39:38.056001+0000","last_became_active":"2023-10-30T12:18:01.393697+0000","last_became_peered":"2023-10-30T12:18:01.393697+0000","last_unstale":"2023-10-30T12:39:38.056001+0000","last_undegraded":"2023-10-30T12:39:38.056001+0000","last_fullsized":"2023-10-30T12:39:38.056001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:15:29.710122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":941,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.832524+0000","last_change":"2023-10-30T12:17:56.604947+0000","last_active":"2023-10-30T12:39:41.832524+0000","last_peered":"2023-10-30T12:39:41.832524+0000","last_clean":"2023-10-30T12:39:41.832524+0000","last_became_active":"2023-10-30T12:17:56.575838+0000","last_became_peered":"2023-10-30T12:17:56.575838+0000","last_unstale":"2023-10-30T12:39:41.832524+0000","last_undegraded":"2023-10-30T12:39:41.832524+0000","last_fullsized":"2023-10-30T12:39:41.832524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_clean_scrub_stamp":"2023-10-30T12:17:53.592563+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:47.623228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":845,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.080335+0000","last_change":"2023-10-30T12:19:25.736207+0000","last_active":"2023-10-30T12:39:38.080335+0000","last_peered":"2023-10-30T12:39:38.080335+0000","last_clean":"2023-10-30T12:39:38.080335+0000","last_became_active":"2023-10-30T12:19:25.715885+0000","last_became_peered":"2023-10-30T12:19:25.715885+0000","last_unstale":"2023-10-30T12:39:38.080335+0000","last_undegraded":"2023-10-30T12:39:38.080335+0000","last_fullsized":"2023-10-30T12:39:38.080335+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:50.517927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":838,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:39.037645+0000","last_change":"2023-10-30T12:19:29.928614+0000","last_active":"2023-10-30T12:39:39.037645+0000","last_peered":"2023-10-30T12:39:39.037645+0000","last_clean":"2023-10-30T12:39:39.037645+0000","last_became_active":"2023-10-30T12:19:29.879226+0000","last_became_peered":"2023-10-30T12:19:29.879226+0000","last_unstale":"2023-10-30T12:39:39.037645+0000","last_undegraded":"2023-10-30T12:39:39.037645+0000","last_fullsized":"2023-10-30T12:39:39.037645+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:26:19.751528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":843,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.068307+0000","last_change":"2023-10-30T12:19:40.799718+0000","last_active":"2023-10-30T12:39:38.068307+0000","last_peered":"2023-10-30T12:39:38.068307+0000","last_clean":"2023-10-30T12:39:38.068307+0000","last_became_active":"2023-10-30T12:19:40.794341+0000","last_became_peered":"2023-10-30T12:19:40.794341+0000","last_unstale":"2023-10-30T12:39:38.068307+0000","last_undegraded":"2023-10-30T12:39:38.068307+0000","last_fullsized":"2023-10-30T12:39:38.068307+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:58.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":844,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.077563+0000","last_change":"2023-10-30T12:19:28.247323+0000","last_active":"2023-10-30T12:39:38.077563+0000","last_peered":"2023-10-30T12:39:38.077563+0000","last_clean":"2023-10-30T12:39:38.077563+0000","last_became_active":"2023-10-30T12:19:28.238513+0000","last_became_peered":"2023-10-30T12:19:28.238513+0000","last_unstale":"2023-10-30T12:39:38.077563+0000","last_undegraded":"2023-10-30T12:39:38.077563+0000","last_fullsized":"2023-10-30T12:39:38.077563+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:43.483993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":836,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:39:42.057559+0000","last_change":"2023-10-30T12:19:27.303856+0000","last_active":"2023-10-30T12:39:42.057559+0000","last_peered":"2023-10-30T12:39:42.057559+0000","last_clean":"2023-10-30T12:39:42.057559+0000","last_became_active":"2023-10-30T12:19:27.284772+0000","last_became_peered":"2023-10-30T12:19:27.284772+0000","last_unstale":"2023-10-30T12:39:42.057559+0000","last_undegraded":"2023-10-30T12:39:42.057559+0000","last_fullsized":"2023-10-30T12:39:42.057559+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:40.340431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":827,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.834340+0000","last_change":"2023-10-30T12:19:33.492307+0000","last_active":"2023-10-30T12:39:41.834340+0000","last_peered":"2023-10-30T12:39:41.834340+0000","last_clean":"2023-10-30T12:39:41.834340+0000","last_became_active":"2023-10-30T12:19:33.483122+0000","last_became_peered":"2023-10-30T12:19:33.483122+0000","last_unstale":"2023-10-30T12:39:41.834340+0000","last_undegraded":"2023-10-30T12:39:41.834340+0000","last_fullsized":"2023-10-30T12:39:41.834340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:16.210880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":839,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.998483+0000","last_change":"2023-10-30T12:19:26.678316+0000","last_active":"2023-10-30T12:39:38.998483+0000","last_peered":"2023-10-30T12:39:38.998483+0000","last_clean":"2023-10-30T12:39:38.998483+0000","last_became_active":"2023-10-30T12:19:26.663005+0000","last_became_peered":"2023-10-30T12:19:26.663005+0000","last_unstale":"2023-10-30T12:39:38.998483+0000","last_undegraded":"2023-10-30T12:39:38.998483+0000","last_fullsized":"2023-10-30T12:39:38.998483+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:22:45.623416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":844,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:39:38.011668+0000","last_change":"2023-10-30T12:19:26.773408+0000","last_active":"2023-10-30T12:39:38.011668+0000","last_peered":"2023-10-30T12:39:38.011668+0000","last_clean":"2023-10-30T12:39:38.011668+0000","last_became_active":"2023-10-30T12:19:26.766234+0000","last_became_peered":"2023-10-30T12:19:26.766234+0000","last_unstale":"2023-10-30T12:39:38.011668+0000","last_undegraded":"2023-10-30T12:39:38.011668+0000","last_fullsized":"2023-10-30T12:39:38.011668+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:39.619411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":836,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:39:42.445361+0000","last_change":"2023-10-30T12:19:29.219323+0000","last_active":"2023-10-30T12:39:42.445361+0000","last_peered":"2023-10-30T12:39:42.445361+0000","last_clean":"2023-10-30T12:39:42.445361+0000","last_became_active":"2023-10-30T12:19:29.191499+0000","last_became_peered":"2023-10-30T12:19:29.191499+0000","last_unstale":"2023-10-30T12:39:42.445361+0000","last_undegraded":"2023-10-30T12:39:42.445361+0000","last_fullsized":"2023-10-30T12:39:42.445361+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.259618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":848,"reported_epoch":896,"state":"active+clean","last_fresh":"2023-10-30T12:39:41.270235+0000","last_change":"2023-10-30T12:19:30.420980+0000","last_active":"2023-10-30T12:39:41.270235+0000","last_peered":"2023-10-30T12:39:41.270235+0000","last_clean":"2023-10-30T12:39:41.270235+0000","last_became_active":"2023-10-30T12:19:30.408141+0000","last_became_peered":"2023-10-30T12:19:30.408141+0000","last_unstale":"2023-10-30T12:39:41.270235+0000","last_undegraded":"2023-10-30T12:39:41.270235+0000","last_fullsized":"2023-10-30T12:39:41.270235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:04:46.816451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":834,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.223908+0000","last_change":"2023-10-30T12:19:40.539149+0000","last_active":"2023-10-30T12:39:40.223908+0000","last_peered":"2023-10-30T12:39:40.223908+0000","last_clean":"2023-10-30T12:39:40.223908+0000","last_became_active":"2023-10-30T12:19:40.528642+0000","last_became_peered":"2023-10-30T12:19:40.528642+0000","last_unstale":"2023-10-30T12:39:40.223908+0000","last_undegraded":"2023-10-30T12:39:40.223908+0000","last_fullsized":"2023-10-30T12:39:40.223908+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:34.653320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":835,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:39:40.295847+0000","last_change":"2023-10-30T12:19:30.748646+0000","last_active":"2023-10-30T12:39:40.295847+0000","last_peered":"2023-10-30T12:39:40.295847+0000","last_clean":"2023-10-30T12:39:40.295847+0000","last_became_active":"2023-10-30T12:19:30.720900+0000","last_became_peered":"2023-10-30T12:19:30.720900+0000","last_unstale":"2023-10-30T12:39:40.295847+0000","last_undegraded":"2023-10-30T12:39:40.295847+0000","last_fullsized":"2023-10-30T12:39:40.295847+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:59.574207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509705,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63332,"kb_used_data":9304,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308508,"statfs":{"total":96636764160,"available":96571912192,"internally_reserved":0,"allocated":9527296,"data_stored":4823141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":23.231999999999999,"5min":17.655999999999999,"15min":16.648},"min":{"1min":6.4530000000000003,"5min":4.0250000000000004,"15min":4.0250000000000004},"max":{"1min":181.905,"5min":233.52500000000001,"15min":233.52500000000001},"last":30.998999999999999},{"interface":"front","average":{"1min":27.143000000000001,"5min":21.59,"15min":22.366},"min":{"1min":9.7650000000000006,"5min":5.6740000000000004,"15min":5.6740000000000004},"max":{"1min":183.072,"5min":214.31899999999999,"15min":365.47399999999999},"last":35.723999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.035,"5min":12.183,"15min":14.696},"min":{"1min":6.0800000000000001,"5min":4.4359999999999999,"15min":4.2389999999999999},"max":{"1min":63.286999999999999,"5min":63.286999999999999,"15min":142.51400000000001},"last":6.6500000000000004},{"interface":"front","average":{"1min":17.978999999999999,"5min":14.823,"15min":17.242000000000001},"min":{"1min":6.056,"5min":5.1689999999999996,"15min":5.1669999999999998},"max":{"1min":61.552,"5min":112.276,"15min":211.11600000000001},"last":7.8440000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.715,"5min":21.622,"15min":20.561},"min":{"1min":6.5259999999999998,"5min":6.2530000000000001,"15min":6.2530000000000001},"max":{"1min":61.927,"5min":187.00399999999999,"15min":187.00399999999999},"last":9.4510000000000005},{"interface":"front","average":{"1min":20.795999999999999,"5min":23.681999999999999,"15min":21.565000000000001},"min":{"1min":8.2710000000000008,"5min":6.5359999999999996,"15min":5.6260000000000003},"max":{"1min":80.686999999999998,"5min":240.99799999999999,"15min":240.99799999999999},"last":8.8460000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":20.48,"5min":26.420000000000002,"15min":25.207999999999998},"min":{"1min":10.49,"5min":7.5990000000000002,"15min":6.2240000000000002},"max":{"1min":62.517000000000003,"5min":279.94099999999997,"15min":279.94099999999997},"last":12.83},{"interface":"front","average":{"1min":17.039000000000001,"5min":24.077999999999999,"15min":23.585000000000001},"min":{"1min":8.1370000000000005,"5min":6.617,"15min":6.5960000000000001},"max":{"1min":55.351999999999997,"5min":180.74600000000001,"15min":180.74600000000001},"last":36.521999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":13.237,"5min":19.042000000000002,"15min":18.274999999999999},"min":{"1min":6.617,"5min":6.617,"15min":6.1840000000000002},"max":{"1min":59.103000000000002,"5min":184.88,"15min":184.88},"last":31.678000000000001},{"interface":"front","average":{"1min":20.768999999999998,"5min":26.350000000000001,"15min":25.687999999999999},"min":{"1min":11.442,"5min":9.4160000000000004,"15min":7.3540000000000001},"max":{"1min":54.426000000000002,"5min":374.38900000000001,"15min":374.38900000000001},"last":13.537000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":19.882000000000001,"5min":25.530999999999999,"15min":25.718},"min":{"1min":10.507,"5min":8.1210000000000004,"15min":6.2729999999999997},"max":{"1min":43.252000000000002,"5min":218.28800000000001,"15min":218.28800000000001},"last":39.472999999999999},{"interface":"front","average":{"1min":17.363,"5min":25.651,"15min":23.794},"min":{"1min":6.7160000000000002,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":34.281999999999996,"5min":195.74700000000001,"15min":195.74700000000001},"last":32.414999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":21.611999999999998,"5min":23.324000000000002,"15min":27.591999999999999},"min":{"1min":9.8620000000000001,"5min":9.593,"15min":8.3320000000000007},"max":{"1min":69.704999999999998,"5min":178.68100000000001,"15min":224.71000000000001},"last":65.272000000000006},{"interface":"front","average":{"1min":19.506,"5min":24.533000000000001,"15min":25.074000000000002},"min":{"1min":9.7799999999999994,"5min":7.2469999999999999,"15min":7.2469999999999999},"max":{"1min":39.325000000000003,"5min":187.14500000000001,"15min":210.05199999999999},"last":33.552999999999997}]}]},{"osd":5,"up_from":15,"seq":64424509705,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82020,"kb_used_data":10132,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289820,"statfs":{"total":96636764160,"available":96552775680,"internally_reserved":0,"allocated":10375168,"data_stored":5538149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":82,"apply_latency_ms":82,"commit_latency_ns":82000000,"apply_latency_ns":82000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:38:43 2023","interfaces":[{"interface":"back","average":{"1min":22.297000000000001,"5min":15.888999999999999,"15min":28.812000000000001},"min":{"1min":4.4119999999999999,"5min":4.1050000000000004,"15min":4.1050000000000004},"max":{"1min":136.36000000000001,"5min":158.87799999999999,"15min":627.18200000000002},"last":12.56},{"interface":"front","average":{"1min":18.611999999999998,"5min":16.850000000000001,"15min":26.960999999999999},"min":{"1min":6.367,"5min":4.7469999999999999,"15min":4.7469999999999999},"max":{"1min":94.358999999999995,"5min":146.374,"15min":341.17000000000002},"last":7.75}]},{"osd":1,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":18.443999999999999,"5min":16.146000000000001,"15min":28.567},"min":{"1min":5.8040000000000003,"5min":5.5300000000000002,"15min":5.5300000000000002},"max":{"1min":138.964,"5min":138.964,"15min":620.21500000000003},"last":4.9160000000000004},{"interface":"front","average":{"1min":17.212,"5min":17.853000000000002,"15min":27.408000000000001},"min":{"1min":7.1970000000000001,"5min":5.6150000000000002,"15min":5.4109999999999996},"max":{"1min":93.744,"5min":162.25200000000001,"15min":334.62},"last":13.255000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":18.535,"5min":18.928000000000001,"15min":23.741},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":134.77000000000001,"5min":208.28100000000001,"15min":246.37},"last":8.5500000000000007},{"interface":"front","average":{"1min":21.039000000000001,"5min":20.257999999999999,"15min":24.73},"min":{"1min":8.4540000000000006,"5min":6.3049999999999997,"15min":5.2770000000000001},"max":{"1min":141.99799999999999,"5min":141.99799999999999,"15min":348.41000000000003},"last":13.888}]},{"osd":3,"last update":"Mon Oct 30 12:39:26 2023","interfaces":[{"interface":"back","average":{"1min":19.34,"5min":23.516999999999999,"15min":25.564},"min":{"1min":6.8339999999999996,"5min":6.3860000000000001,"15min":6.3860000000000001},"max":{"1min":97.822999999999993,"5min":148.125,"15min":273.10300000000001},"last":16.361999999999998},{"interface":"front","average":{"1min":24.561,"5min":21.800000000000001,"15min":24.858000000000001},"min":{"1min":9.3970000000000002,"5min":6.2889999999999997,"15min":5.8029999999999999},"max":{"1min":94.296000000000006,"5min":154.857,"15min":282.28500000000003},"last":14.513}]},{"osd":4,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":25.477,"5min":23.068000000000001,"15min":33.439},"min":{"1min":11.792999999999999,"5min":9.5109999999999992,"15min":7.0259999999999998},"max":{"1min":144.102,"5min":144.102,"15min":512.01199999999994},"last":15.071999999999999},{"interface":"front","average":{"1min":27.440999999999999,"5min":25.501000000000001,"15min":31.102},"min":{"1min":7.4530000000000003,"5min":7.4530000000000003,"15min":6.3890000000000002},"max":{"1min":113.208,"5min":169.45500000000001,"15min":386.62200000000001},"last":9.8640000000000008}]},{"osd":6,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":28.742000000000001,"5min":28.922000000000001,"15min":31.957000000000001},"min":{"1min":14.225,"5min":11.138,"15min":7.9729999999999999},"max":{"1min":144.726,"5min":242.16800000000001,"15min":588.93899999999996},"last":15.701000000000001},{"interface":"front","average":{"1min":32.106999999999999,"5min":34.963999999999999,"15min":37.119},"min":{"1min":16.084,"5min":12.661,"15min":7.25},"max":{"1min":161.18000000000001,"5min":319.17700000000002,"15min":500.173},"last":17.097999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:39:26 2023","interfaces":[{"interface":"back","average":{"1min":20.66,"5min":27.561,"15min":28.265999999999998},"min":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"max":{"1min":113.35899999999999,"5min":258.61200000000002,"15min":258.61200000000002},"last":10.956},{"interface":"front","average":{"1min":16.838999999999999,"5min":22.385999999999999,"15min":24.34},"min":{"1min":6.109,"5min":6.109,"15min":6.109},"max":{"1min":55.759,"5min":145.15000000000001,"15min":245.71899999999999},"last":11.893000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509706,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62520,"kb_used_data":8940,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309320,"statfs":{"total":96636764160,"available":96572743680,"internally_reserved":0,"allocated":9154560,"data_stored":4578828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":8.8330000000000002,"5min":15.872,"15min":22.242999999999999},"min":{"1min":4.7569999999999997,"5min":4.2140000000000004,"15min":4.0620000000000003},"max":{"1min":16.516999999999999,"5min":158.499,"15min":295.56400000000002},"last":16.530000000000001},{"interface":"front","average":{"1min":8.3000000000000007,"5min":18.66,"15min":23.565999999999999},"min":{"1min":4.4080000000000004,"5min":4.4080000000000004,"15min":4.4080000000000004},"max":{"1min":14.081,"5min":182.96600000000001,"15min":368.01900000000001},"last":7.133}]},{"osd":1,"last update":"Mon Oct 30 12:38:45 2023","interfaces":[{"interface":"back","average":{"1min":11.939,"5min":17.212,"15min":23.172000000000001},"min":{"1min":5.9829999999999997,"5min":5.6420000000000003,"15min":4.5389999999999997},"max":{"1min":53.116,"5min":171.24100000000001,"15min":322.97800000000001},"last":17.254999999999999},{"interface":"front","average":{"1min":17.463999999999999,"5min":20.486999999999998,"15min":22.018000000000001},"min":{"1min":6.9050000000000002,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":105.38800000000001,"5min":168.833,"15min":245.44900000000001},"last":28.896999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:38:51 2023","interfaces":[{"interface":"back","average":{"1min":18.649999999999999,"5min":26.581,"15min":25.079000000000001},"min":{"1min":7.9740000000000002,"5min":6.0039999999999996,"15min":6.0039999999999996},"max":{"1min":112.88,"5min":316.74799999999999,"15min":316.74799999999999},"last":8.1340000000000003},{"interface":"front","average":{"1min":16.074999999999999,"5min":26.199999999999999,"15min":26.724},"min":{"1min":8.0690000000000008,"5min":5.9489999999999998,"15min":5.9489999999999998},"max":{"1min":53.756,"5min":317.55399999999997,"15min":317.55399999999997},"last":29.728000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":14.321999999999999,"5min":23.728999999999999,"15min":28.404},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":6.7270000000000003},"max":{"1min":21.91,"5min":161.197,"15min":383.87099999999998},"last":26.135000000000002},{"interface":"front","average":{"1min":13.757999999999999,"5min":26.026,"15min":26.616},"min":{"1min":8.5,"5min":8.5,"15min":6.6529999999999996},"max":{"1min":18.109999999999999,"5min":260.94200000000001,"15min":284.61700000000002},"last":30.518999999999998}]},{"osd":5,"last update":"Mon Oct 30 12:38:45 2023","interfaces":[{"interface":"back","average":{"1min":23.707999999999998,"5min":27.972000000000001,"15min":30.349},"min":{"1min":7.9429999999999996,"5min":7.9429999999999996,"15min":5.7119999999999997},"max":{"1min":183.55000000000001,"5min":185.47200000000001,"15min":261.80200000000002},"last":31.138999999999999},{"interface":"front","average":{"1min":17.798999999999999,"5min":22.954999999999998,"15min":29.309000000000001},"min":{"1min":9.1099999999999994,"5min":6.3760000000000003,"15min":6.3760000000000003},"max":{"1min":102.428,"5min":174.38999999999999,"15min":371.35700000000003},"last":33.390999999999998}]},{"osd":6,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":23.914999999999999,"5min":27.789999999999999,"15min":27.952999999999999},"min":{"1min":10.460000000000001,"5min":9.4939999999999998,"15min":8.0060000000000002},"max":{"1min":95.795000000000002,"5min":215.31800000000001,"15min":312.44999999999999},"last":34.624000000000002},{"interface":"front","average":{"1min":19.725999999999999,"5min":28.399999999999999,"15min":27.664000000000001},"min":{"1min":9.859,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":63.609999999999999,"5min":266.50599999999997,"15min":320.50999999999999},"last":27.367000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":17.331,"5min":26.408000000000001,"15min":28.349},"min":{"1min":9.048,"5min":9.048,"15min":7.9370000000000003},"max":{"1min":55.015000000000001,"5min":193.95400000000001,"15min":319.84199999999998},"last":37.808},{"interface":"front","average":{"1min":15.836,"5min":26.617000000000001,"15min":27.077000000000002},"min":{"1min":10.515000000000001,"5min":8.9489999999999998,"15min":8.5410000000000004},"max":{"1min":28.527000000000001,"5min":252.232,"15min":315.06900000000002},"last":35.856000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509703,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85592,"kb_used_data":9768,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286248,"statfs":{"total":96636764160,"available":96549117952,"internally_reserved":0,"allocated":10002432,"data_stored":5188384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":82,"apply_latency_ms":82,"commit_latency_ns":82000000,"apply_latency_ns":82000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.731999999999999,"5min":21.190000000000001,"15min":21.754999999999999},"min":{"1min":4.609,"5min":4.2460000000000004,"15min":4.1159999999999997},"max":{"1min":210.398,"5min":303.58600000000001,"15min":303.58600000000001},"last":6.9340000000000002},{"interface":"front","average":{"1min":26.295999999999999,"5min":17.559999999999999,"15min":21.213000000000001},"min":{"1min":6.0049999999999999,"5min":4.8819999999999997,"15min":4.1070000000000002},"max":{"1min":187.82400000000001,"5min":187.82400000000001,"15min":268.68799999999999},"last":13.012}]},{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":42.889000000000003,"5min":25.684999999999999,"15min":23.533999999999999},"min":{"1min":7.7990000000000004,"5min":5.6070000000000002,"15min":4.8940000000000001},"max":{"1min":220.96600000000001,"5min":220.96600000000001,"15min":389.303},"last":12.266999999999999},{"interface":"front","average":{"1min":43.094999999999999,"5min":25.721,"15min":24.445},"min":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":5.0890000000000004},"max":{"1min":222.328,"5min":222.328,"15min":468.13299999999998},"last":6.2850000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":19.759,"5min":21.931000000000001,"15min":22.350999999999999},"min":{"1min":8.0250000000000004,"5min":6.8230000000000004,"15min":5.6150000000000002},"max":{"1min":135.97399999999999,"5min":352.09399999999999,"15min":352.09399999999999},"last":9.6430000000000007},{"interface":"front","average":{"1min":18.385000000000002,"5min":20.425999999999998,"15min":23.646999999999998},"min":{"1min":7.7889999999999997,"5min":6.9850000000000003,"15min":6.9850000000000003},"max":{"1min":94.552999999999997,"5min":184.04599999999999,"15min":287.33699999999999},"last":17.280999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.673000000000002,"5min":25.640000000000001,"15min":28.405000000000001},"min":{"1min":8.6929999999999996,"5min":7.1799999999999997,"15min":6.5010000000000003},"max":{"1min":211.03200000000001,"5min":310.12099999999998,"15min":310.12099999999998},"last":8.2880000000000003},{"interface":"front","average":{"1min":31.611000000000001,"5min":25.715,"15min":25.268000000000001},"min":{"1min":9.2690000000000001,"5min":6.7169999999999996,"15min":6.0919999999999996},"max":{"1min":187.15700000000001,"5min":359.87799999999999,"15min":359.87799999999999},"last":8.9420000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":44.430999999999997,"5min":29.558,"15min":30.042999999999999},"min":{"1min":9.8840000000000003,"5min":9,"15min":6.851},"max":{"1min":274.15300000000002,"5min":360.88099999999997,"15min":564.26700000000005},"last":10.279},{"interface":"front","average":{"1min":50.723999999999997,"5min":28.849,"15min":28.324999999999999},"min":{"1min":10.619,"5min":8.8680000000000003,"15min":8.0109999999999992},"max":{"1min":289.68099999999998,"5min":289.68099999999998,"15min":294.392},"last":13.667999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":30.876999999999999,"5min":29.062999999999999,"15min":30.353999999999999},"min":{"1min":11.016,"5min":9.3369999999999997,"15min":8.8149999999999995},"max":{"1min":197.09200000000001,"5min":374.33999999999997,"15min":374.33999999999997},"last":14.846},{"interface":"front","average":{"1min":33.301000000000002,"5min":31.440000000000001,"15min":31.552},"min":{"1min":11.416,"5min":11.416,"15min":8.9079999999999995},"max":{"1min":177.63499999999999,"5min":352.79500000000002,"15min":352.79500000000002},"last":15.577}]},{"osd":7,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":36.420000000000002,"5min":32.140999999999998,"15min":34.933},"min":{"1min":12.868,"5min":11.401999999999999,"15min":8.5099999999999998},"max":{"1min":203.70099999999999,"5min":382.68299999999999,"15min":382.68299999999999},"last":16.646000000000001},{"interface":"front","average":{"1min":24.998999999999999,"5min":29.645,"15min":32.134},"min":{"1min":11.976000000000001,"5min":9.0850000000000009,"15min":8.7699999999999996},"max":{"1min":86.024000000000001,"5min":361.47199999999998,"15min":562.05700000000002},"last":18.204999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509704,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67080,"kb_used_data":8736,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94304760,"statfs":{"total":96636764160,"available":96568074240,"internally_reserved":0,"allocated":8945664,"data_stored":4449339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:02 2023","interfaces":[{"interface":"back","average":{"1min":9.0120000000000005,"5min":15.948,"15min":19.835000000000001},"min":{"1min":4.4260000000000002,"5min":4.4260000000000002,"15min":3.8079999999999998},"max":{"1min":20.417000000000002,"5min":121.476,"15min":337.86700000000002},"last":6.0289999999999999},{"interface":"front","average":{"1min":10.103,"5min":14.271000000000001,"15min":18.513000000000002},"min":{"1min":5.6310000000000002,"5min":4.4279999999999999,"15min":4.1399999999999997},"max":{"1min":18.503,"5min":119.87,"15min":343.983},"last":7.266}]},{"osd":1,"last update":"Mon Oct 30 12:38:53 2023","interfaces":[{"interface":"back","average":{"1min":10.884,"5min":16.09,"15min":23.937000000000001},"min":{"1min":7.0860000000000003,"5min":4.9790000000000001,"15min":4.6559999999999997},"max":{"1min":16.172000000000001,"5min":153.751,"15min":501.19900000000001},"last":23.221},{"interface":"front","average":{"1min":13.513999999999999,"5min":18.212,"15min":22.693999999999999},"min":{"1min":6.9859999999999998,"5min":6.9859999999999998,"15min":6.649},"max":{"1min":34.274000000000001,"5min":122.767,"15min":391.58300000000003},"last":12.244999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:38:57 2023","interfaces":[{"interface":"back","average":{"1min":13.710000000000001,"5min":21.341999999999999,"15min":27.303000000000001},"min":{"1min":7.7140000000000004,"5min":6.923,"15min":6.6379999999999999},"max":{"1min":34.911999999999999,"5min":173.56899999999999,"15min":414.69299999999998},"last":20.879000000000001},{"interface":"front","average":{"1min":14.257,"5min":21.814,"15min":28.373000000000001},"min":{"1min":7.484,"5min":7.1769999999999996,"15min":6.6269999999999998},"max":{"1min":32.726999999999997,"5min":197.904,"15min":502.19299999999998},"last":23.837}]},{"osd":4,"last update":"Mon Oct 30 12:39:42 2023","interfaces":[{"interface":"back","average":{"1min":16.821999999999999,"5min":22.812000000000001,"15min":23.850999999999999},"min":{"1min":8,"5min":8,"15min":6.415},"max":{"1min":40.204999999999998,"5min":157.77199999999999,"15min":331.834},"last":29.629999999999999},{"interface":"front","average":{"1min":15.441000000000001,"5min":17.044,"15min":18.234999999999999},"min":{"1min":6.8239999999999998,"5min":6.1509999999999998,"15min":5.984},"max":{"1min":47.32,"5min":134.452,"15min":179.89099999999999},"last":6.8239999999999998}]},{"osd":5,"last update":"Mon Oct 30 12:38:42 2023","interfaces":[{"interface":"back","average":{"1min":16.175000000000001,"5min":25.510999999999999,"15min":34.662999999999997},"min":{"1min":8.6720000000000006,"5min":8.0619999999999994,"15min":7.4500000000000002},"max":{"1min":36.591000000000001,"5min":245.22800000000001,"15min":695.71100000000001},"last":42.088000000000001},{"interface":"front","average":{"1min":18.864999999999998,"5min":27.582000000000001,"15min":36.226999999999997},"min":{"1min":12.657,"5min":9.6349999999999998,"15min":9.3930000000000007},"max":{"1min":37.334000000000003,"5min":241.41999999999999,"15min":578.82500000000005},"last":34.076999999999998}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":16.959,"5min":19.983000000000001,"15min":20.140000000000001},"min":{"1min":8.1110000000000007,"5min":6.3239999999999998,"15min":6.1100000000000003},"max":{"1min":49.598999999999997,"5min":146.75,"15min":182.97499999999999},"last":10.009},{"interface":"front","average":{"1min":20.119,"5min":23.864000000000001,"15min":25.939},"min":{"1min":11.311,"5min":9.8819999999999997,"15min":8.7729999999999997},"max":{"1min":45.713000000000001,"5min":154.619,"15min":315.71100000000001},"last":33.064999999999998}]},{"osd":7,"last update":"Mon Oct 30 12:38:53 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":18.695,"15min":22.042999999999999},"min":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"max":{"1min":27.41,"5min":123.926,"15min":228.67400000000001},"last":9.4429999999999996},{"interface":"front","average":{"1min":19.202000000000002,"5min":25.521000000000001,"15min":29.041},"min":{"1min":11.582000000000001,"5min":10.462999999999999,"15min":9.0589999999999993},"max":{"1min":38.158000000000001,"5min":160.93600000000001,"15min":328.51799999999997},"last":34.829000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509704,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63800,"kb_used_data":10184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308040,"statfs":{"total":96636764160,"available":96571432960,"internally_reserved":0,"allocated":10428416,"data_stored":5595488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":10.207000000000001,"5min":11.294,"15min":18.631},"min":{"1min":5.5519999999999996,"5min":4.3760000000000003,"15min":4.3760000000000003},"max":{"1min":24.184999999999999,"5min":75.006,"15min":294.67099999999999},"last":14.157},{"interface":"front","average":{"1min":11.981999999999999,"5min":11.672000000000001,"15min":17.254000000000001},"min":{"1min":6.4210000000000003,"5min":5.2400000000000002,"15min":4.952},"max":{"1min":27.530999999999999,"5min":61.932000000000002,"15min":241.066},"last":7.1820000000000004}]},{"osd":2,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":16.41,"5min":14.955,"15min":19.184000000000001},"min":{"1min":6.4299999999999997,"5min":6.4299999999999997,"15min":6.0519999999999996},"max":{"1min":67.140000000000001,"5min":75.491,"15min":295.50599999999997},"last":8.3179999999999996},{"interface":"front","average":{"1min":18.949999999999999,"5min":13.51,"15min":19.122},"min":{"1min":6.8810000000000002,"5min":6.1580000000000004,"15min":5.327},"max":{"1min":135.00800000000001,"5min":135.00800000000001,"15min":269.99200000000002},"last":11.167999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":19.390000000000001,"15min":21.367999999999999},"min":{"1min":6.9889999999999999,"5min":6.4779999999999998,"15min":6.4779999999999998},"max":{"1min":25.774999999999999,"5min":310.40800000000002,"15min":551.58000000000004},"last":9.5630000000000006},{"interface":"front","average":{"1min":17.298999999999999,"5min":28.062000000000001,"15min":27.957000000000001},"min":{"1min":9.8819999999999997,"5min":9.8819999999999997,"15min":9.1999999999999993},"max":{"1min":29.579000000000001,"5min":338.88600000000002,"15min":562.06899999999996},"last":15.07}]},{"osd":4,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":14.186,"5min":16.617000000000001,"15min":21.454000000000001},"min":{"1min":7.0469999999999997,"5min":6.7480000000000002,"15min":6.6109999999999998},"max":{"1min":40.465000000000003,"5min":84.212000000000003,"15min":285.26499999999999},"last":20.274000000000001},{"interface":"front","average":{"1min":14.521000000000001,"5min":16.108000000000001,"15min":22.126000000000001},"min":{"1min":9.0500000000000007,"5min":6.7450000000000001,"15min":6.7450000000000001},"max":{"1min":42.087000000000003,"5min":62.616,"15min":327.59699999999998},"last":16.891999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":35.115000000000002,"5min":25.553999999999998,"15min":26.443999999999999},"min":{"1min":9.9410000000000007,"5min":8.1639999999999997,"15min":7.3070000000000004},"max":{"1min":322.59899999999999,"5min":322.59899999999999,"15min":322.59899999999999},"last":16.023},{"interface":"front","average":{"1min":32.095999999999997,"5min":21.616,"15min":24.370999999999999},"min":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"max":{"1min":283.06,"5min":283.06,"15min":286.43799999999999},"last":12.630000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":13.723000000000001,"5min":17.039000000000001,"15min":22.716000000000001},"min":{"1min":7.6059999999999999,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":28.151,"5min":71.662999999999997,"15min":348.91000000000003},"last":10.596},{"interface":"front","average":{"1min":18.815999999999999,"5min":23.582999999999998,"15min":26.449000000000002},"min":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":8.4819999999999993},"max":{"1min":33.466000000000001,"5min":99.231999999999999,"15min":301.99799999999999},"last":19.335000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":21.850999999999999,"5min":21.949000000000002,"15min":25.800999999999998},"min":{"1min":13.108000000000001,"5min":9.2780000000000005,"15min":7.8739999999999997},"max":{"1min":68.974999999999994,"5min":107.703,"15min":305.19900000000001},"last":17.609000000000002},{"interface":"front","average":{"1min":20.207000000000001,"5min":20.809000000000001,"15min":26.821000000000002},"min":{"1min":8.8640000000000008,"5min":8.8640000000000008,"15min":8.0920000000000005},"max":{"1min":70.406999999999996,"5min":106.678,"15min":277.05200000000002},"last":11.855}]}]},{"osd":0,"up_from":15,"seq":64424509704,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63140,"kb_used_data":9544,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308700,"statfs":{"total":96636764160,"available":96572108800,"internally_reserved":0,"allocated":9773056,"data_stored":5007107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":14.013,"5min":16.620999999999999,"15min":19.768000000000001},"min":{"1min":6.117,"5min":4.1799999999999997,"15min":4.1799999999999997},"max":{"1min":73.113,"5min":232.798,"15min":287.096},"last":47.450000000000003},{"interface":"front","average":{"1min":11.015000000000001,"5min":14.513,"15min":22.515999999999998},"min":{"1min":7.6879999999999997,"5min":5.2370000000000001,"15min":4.7590000000000003},"max":{"1min":29.097000000000001,"5min":218.61099999999999,"15min":425.762},"last":44.664999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:39:40 2023","interfaces":[{"interface":"back","average":{"1min":21.314,"5min":17.907,"15min":20.555},"min":{"1min":8.3599999999999994,"5min":5.2480000000000002,"15min":5.2480000000000002},"max":{"1min":78.947999999999993,"5min":228.90799999999999,"15min":299.46600000000001},"last":78.947999999999993},{"interface":"front","average":{"1min":28.640000000000001,"5min":18.164000000000001,"15min":20.786000000000001},"min":{"1min":6.7779999999999996,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":113.76000000000001,"5min":138.13499999999999,"15min":258.45400000000001},"last":87.332999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.376999999999999,"5min":16.640999999999998,"15min":20.597999999999999},"min":{"1min":9.6329999999999991,"5min":6.5499999999999998,"15min":5.2110000000000003},"max":{"1min":41.456000000000003,"5min":166.21600000000001,"15min":264.97199999999998},"last":55.999000000000002},{"interface":"front","average":{"1min":19.736000000000001,"5min":21.084,"15min":25.367000000000001},"min":{"1min":8.4079999999999995,"5min":8.4079999999999995,"15min":7.218},"max":{"1min":54.859000000000002,"5min":230.73699999999999,"15min":258.93200000000002},"last":58.859000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6620000000000008,"5min":18.376000000000001,"15min":22.655999999999999},"min":{"1min":5.681,"5min":5.681,"15min":5.681},"max":{"1min":24.405999999999999,"5min":209.64599999999999,"15min":209.64599999999999},"last":61.680999999999997},{"interface":"front","average":{"1min":20.053999999999998,"5min":25.309000000000001,"15min":23.466999999999999},"min":{"1min":8.8819999999999997,"5min":8.0990000000000002,"15min":5.8499999999999996},"max":{"1min":90.787999999999997,"5min":208.92400000000001,"15min":208.92400000000001},"last":52.835999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.98,"5min":29.106000000000002,"15min":29.109999999999999},"min":{"1min":10.965,"5min":8.9440000000000008,"15min":7.5510000000000002},"max":{"1min":31.939,"5min":245.16300000000001,"15min":285.57100000000003},"last":60.929000000000002},{"interface":"front","average":{"1min":17.673999999999999,"5min":21.956,"15min":28.129999999999999},"min":{"1min":10.518000000000001,"5min":7.2720000000000002,"15min":7.2720000000000002},"max":{"1min":30.361000000000001,"5min":142.25,"15min":292.07900000000001},"last":54.409999999999997}]},{"osd":6,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":15.827999999999999,"5min":20.135999999999999,"15min":21.259},"min":{"1min":7.7000000000000002,"5min":7.4370000000000003,"15min":7.4370000000000003},"max":{"1min":43.283999999999999,"5min":171.99600000000001,"15min":236.40000000000001},"last":60.557000000000002},{"interface":"front","average":{"1min":18.968,"5min":24.116,"15min":25.454999999999998},"min":{"1min":12.252000000000001,"5min":10.470000000000001,"15min":8.9199999999999999},"max":{"1min":31.385000000000002,"5min":178.59999999999999,"15min":220.68700000000001},"last":65.087000000000003}]},{"osd":7,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":19.300999999999998,"5min":20.318000000000001,"15min":24.036000000000001},"min":{"1min":11.853,"5min":9.2100000000000009,"15min":9.2100000000000009},"max":{"1min":32.664999999999999,"5min":143.49600000000001,"15min":245.87100000000001},"last":55.232999999999997},{"interface":"front","average":{"1min":16.231000000000002,"5min":22.702000000000002,"15min":24.879999999999999},"min":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":32.033000000000001,"5min":181.15100000000001,"15min":260.04000000000002},"last":63.960999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575112,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63560,"kb_used_data":9508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308280,"statfs":{"total":96636764160,"available":96571678720,"internally_reserved":0,"allocated":9736192,"data_stored":4984204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":9.4269999999999996,"5min":10.32,"15min":22.806999999999999},"min":{"1min":4.0960000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"max":{"1min":23.024000000000001,"5min":99.075000000000003,"15min":623.43700000000001},"last":6.4779999999999998},{"interface":"front","average":{"1min":10.904,"5min":11.211,"15min":21.835000000000001},"min":{"1min":4.7279999999999998,"5min":4.202,"15min":4.202},"max":{"1min":24.524999999999999,"5min":51.253999999999998,"15min":563.08900000000006},"last":7.6710000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":9.4900000000000002,"5min":11.772,"15min":19.108000000000001},"min":{"1min":6.04,"5min":5.0019999999999998,"15min":5.0019999999999998},"max":{"1min":13.172000000000001,"5min":68.641999999999996,"15min":276.80399999999997},"last":7.2800000000000002},{"interface":"front","average":{"1min":13.029999999999999,"5min":16.312999999999999,"15min":24.184000000000001},"min":{"1min":9.3659999999999997,"5min":7.008,"15min":7.008},"max":{"1min":18.454000000000001,"5min":80.891999999999996,"15min":289.52800000000002},"last":13.73}]},{"osd":2,"last update":"Mon Oct 30 12:38:51 2023","interfaces":[{"interface":"back","average":{"1min":11.6,"5min":18.939,"15min":22.053999999999998},"min":{"1min":6.1749999999999998,"5min":6.1749999999999998,"15min":5.0529999999999999},"max":{"1min":20.262,"5min":329.81999999999999,"15min":329.81999999999999},"last":10.012},{"interface":"front","average":{"1min":13.555999999999999,"5min":17.236000000000001,"15min":22.600000000000001},"min":{"1min":7.5229999999999997,"5min":6.8129999999999997,"15min":6.0350000000000001},"max":{"1min":18.695,"5min":132.25800000000001,"15min":245.63},"last":15.695}]},{"osd":3,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":14.976000000000001,"5min":19.353999999999999,"15min":25.904},"min":{"1min":8.6329999999999991,"5min":6.4950000000000001,"15min":6.4950000000000001},"max":{"1min":33.936,"5min":97.286000000000001,"15min":185.98400000000001},"last":16.648},{"interface":"front","average":{"1min":17.303000000000001,"5min":21.863,"15min":28.611999999999998},"min":{"1min":8.2799999999999994,"5min":7.7000000000000002,"15min":6.8159999999999998},"max":{"1min":61.064,"5min":247.65899999999999,"15min":361.625},"last":12.743}]},{"osd":4,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":13.592000000000001,"5min":20.183,"15min":29.582000000000001},"min":{"1min":8.7720000000000002,"5min":5.8150000000000004,"15min":5.8150000000000004},"max":{"1min":17.824000000000002,"5min":117.967,"15min":838.78700000000003},"last":17.553000000000001},{"interface":"front","average":{"1min":12.276,"5min":16.821000000000002,"15min":27.617000000000001},"min":{"1min":6.9329999999999998,"5min":6.9329999999999998,"15min":6.6280000000000001},"max":{"1min":18.846,"5min":82.543999999999997,"15min":836.30899999999997},"last":10.99}]},{"osd":5,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":20.395,"5min":22.149000000000001,"15min":32.493000000000002},"min":{"1min":9.3989999999999991,"5min":7.1440000000000001,"15min":6.5110000000000001},"max":{"1min":64.566000000000003,"5min":186.54499999999999,"15min":504.84100000000001},"last":11.638},{"interface":"front","average":{"1min":24.806999999999999,"5min":22.687999999999999,"15min":32.274000000000001},"min":{"1min":11.058999999999999,"5min":7.5919999999999996,"15min":6.7249999999999996},"max":{"1min":108.858,"5min":185.61099999999999,"15min":405.85000000000002},"last":8.6980000000000004}]},{"osd":7,"last update":"Mon Oct 30 12:38:40 2023","interfaces":[{"interface":"back","average":{"1min":16.547999999999998,"5min":21.190999999999999,"15min":28.661999999999999},"min":{"1min":10.946,"5min":9.3940000000000001,"15min":7.5579999999999998},"max":{"1min":20.657,"5min":99.480999999999995,"15min":402.315},"last":18.251999999999999},{"interface":"front","average":{"1min":17.120000000000001,"5min":24.227,"15min":27.654},"min":{"1min":13.159000000000001,"5min":8.8019999999999996,"15min":8.8019999999999996},"max":{"1min":21.486999999999998,"5min":264.99299999999999,"15min":291.69},"last":14.337999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:39:45.680 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:39:45.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.780+0000 7f1c91cb4640 10 monclient: get_monmap_and_config 2023-10-30T12:39:45.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.780+0000 7f1c91cb4640 10 monclient: build_initial_monmap 2023-10-30T12:39:45.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.780+0000 7f1c91cb4640 10 monclient: monmap: 2023-10-30T12:39:45.782 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:45.782 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:45.782 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:45.781078+0000 2023-10-30T12:39:45.782 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:45.781078+0000 2023-10-30T12:39:45.783 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:45.783 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:45.783 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:45.783 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:45.783 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:45.784 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:45.784 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:45.784 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.780+0000 7f1c91cb4640 1 Processor -- start 2023-10-30T12:39:45.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.780+0000 7f1c91cb4640 1 -- start start 2023-10-30T12:39:45.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.780+0000 7f1c91cb4640 10 monclient: init 2023-10-30T12:39:45.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:45.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-10-30T12:39:45.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c8c0b7c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:45.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c141940 0x7f1c8c141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:45.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 10 monclient(hunting): picked mon.noname-b con 0x7f1c8c141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:45.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1c8c13e730 0x7f1c8c13eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:45.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 10 monclient(hunting): picked mon.noname-c con 0x7f1c8c13e730 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:45.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c0a9350 con 0x7f1c8c0b7c10 2023-10-30T12:39:45.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:45.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c0670a0 con 0x7f1c8c13e730 2023-10-30T12:39:45.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:45.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0b7ff0 con 0x7f1c8c141940 2023-10-30T12:39:45.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:45.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c91cb4640 10 monclient(hunting): authenticate will time out at 2877.986330s 2023-10-30T12:39:45.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.781+0000 7f1c82ffd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c141940 0x7f1c8c141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:45.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c8bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1c8c13e730 0x7f1c8c13eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:45.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c82ffd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c141940 0x7f1c8c141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:53924/0 (socket says 172.21.15.148:53924) 2023-10-30T12:39:45.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c82ffd640 1 -- 172.21.15.148:0/4046432983 learned_addr learned my addr 172.21.15.148:0/4046432983 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:45.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c82ffd640 10 monclient(hunting): get_auth_request con 0x7f1c8c141940 auth_method 0 2023-10-30T12:39:45.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:45.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c82ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:45.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c82ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:45.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c8bfff640 10 monclient(hunting): get_auth_request con 0x7f1c8c13e730 auth_method 0 2023-10-30T12:39:45.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:45.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:45.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c8bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:45.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:45.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:45.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.782+0000 7f1c82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:45.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.783+0000 7f1c8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:45.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.783+0000 7f1c8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:45.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.783+0000 7f1c8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:45.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.784+0000 7f1c82ffd640 10 monclient(hunting): handle_auth_done global_id 17693 payload 274 2023-10-30T12:39:45.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.784+0000 7f1c82ffd640 1 -- 172.21.15.148:0/4046432983 >> v2:172.21.15.150:3300/0 conn(0x7f1c8c13e730 msgr2=0x7f1c8c13eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:45.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.784+0000 7f1c82ffd640 1 --2- 172.21.15.148:0/4046432983 >> v2:172.21.15.150:3300/0 conn(0x7f1c8c13e730 0x7f1c8c13eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.784+0000 7f1c82ffd640 1 -- 172.21.15.148:0/4046432983 >> v1:172.21.15.150:6789/0 conn(0x7f1c8c0b7c10 legacy=0x7f1c8c0a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:45.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.784+0000 7f1c82ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:39:45.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.784+0000 7f1c82ffd640 1 monclient: found mon.noname-b 2023-10-30T12:39:45.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.784+0000 7f1c82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:45.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.784+0000 7f1c82ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:45.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.784+0000 7f1c82ffd640 1 -- 172.21.15.148:0/4046432983 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c0aa6a0 con 0x7f1c8c141940 2023-10-30T12:39:45.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.786+0000 7f1c82ffd640 1 --2- 172.21.15.148:0/4046432983 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c141940 0x7f1c8c141d60 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009870 tx=0x7f1c7402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39491450f489245 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:45.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 1 -- 172.21.15.148:0/4046432983 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c740317c0 con 0x7f1c8c141940 2023-10-30T12:39:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 10 monclient: dump: 2023-10-30T12:39:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:45.802 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 10 monclient: _finish_auth 0 2023-10-30T12:39:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 10 monclient: _check_auth_tickets 2023-10-30T12:39:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c91cb4640 5 monclient: authenticate success, global_id 17693 2023-10-30T12:39:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 1 -- 172.21.15.148:0/4046432983 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7402e070 con 0x7f1c8c141940 2023-10-30T12:39:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c91cb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:39:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c91cb4640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 1 -- 172.21.15.148:0/4046432983 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74031b50 con 0x7f1c8c141940 2023-10-30T12:39:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c91cb4640 4 set_mon_vals no callback set 2023-10-30T12:39:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.790+0000 7f1c8affd640 10 monclient: dump: 2023-10-30T12:39:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:45.805 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:45.805 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.805 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.805 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:45.805 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:45.806 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:45.806 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:45.806 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:45.806 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.791+0000 7f1c91cb4640 10 monclient: shutdown 2023-10-30T12:39:45.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.791+0000 7f1c91cb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:45.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.791+0000 7f1c91cb4640 1 -- 172.21.15.148:0/4046432983 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c141940 msgr2=0x7f1c8c141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.791+0000 7f1c91cb4640 1 --2- 172.21.15.148:0/4046432983 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c141940 0x7f1c8c141d60 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009870 tx=0x7f1c7402b860 comp rx=0 tx=0).stop 2023-10-30T12:39:45.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.791+0000 7f1c91cb4640 1 -- 172.21.15.148:0/4046432983 shutdown_connections 2023-10-30T12:39:45.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.791+0000 7f1c91cb4640 1 --2- 172.21.15.148:0/4046432983 >> v2:172.21.15.150:3300/0 conn(0x7f1c8c13e730 0x7f1c8c13eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.791+0000 7f1c91cb4640 1 --2- 172.21.15.148:0/4046432983 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c141940 0x7f1c8c141d60 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.791+0000 7f1c91cb4640 1 -- 172.21.15.148:0/4046432983 >> 172.21.15.148:0/4046432983 conn(0x7f1c8c06c660 msgr2=0x7f1c8c0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:45.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.791+0000 7f1c91cb4640 1 -- 172.21.15.148:0/4046432983 shutdown_connections 2023-10-30T12:39:45.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.791+0000 7f1c91cb4640 1 -- 172.21.15.148:0/4046432983 wait complete. 2023-10-30T12:39:45.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.792+0000 7f1c91cb4640 10 monclient: build_initial_monmap 2023-10-30T12:39:45.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.792+0000 7f1c91cb4640 10 monclient: monmap: 2023-10-30T12:39:45.809 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:45.809 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:45.809 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:45.809 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:45.809 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:45.810 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:45.810 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:45.810 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:45.810 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:45.810 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.792+0000 7f1c91cb4640 1 librados: starting msgr at 2023-10-30T12:39:45.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.792+0000 7f1c91cb4640 1 librados: starting objecter 2023-10-30T12:39:45.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.792+0000 7f1c91cb4640 1 Processor -- start 2023-10-30T12:39:45.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.792+0000 7f1c91cb4640 1 -- start start 2023-10-30T12:39:45.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.792+0000 7f1c91cb4640 1 librados: setting wanted keys 2023-10-30T12:39:45.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.792+0000 7f1c91cb4640 1 librados: calling monclient init 2023-10-30T12:39:45.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.792+0000 7f1c91cb4640 10 monclient: init 2023-10-30T12:39:45.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:45.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:39:45.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c8c0b7c10 0x7f1c8c0b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:45.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 10 monclient(hunting): picked mon.noname-b con 0x7f1c8c0b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:45.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c13e730 0x7f1c8c11f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:45.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 10 monclient(hunting): picked mon.noname-c con 0x7f1c8c13e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:45.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c8c141940 0x7f1c8c1d7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:45.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 10 monclient(hunting): picked mon.noname-a con 0x7f1c8c141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:45.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:45.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c13f530 con 0x7f1c8c141940 2023-10-30T12:39:45.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:45.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0affb0 con 0x7f1c8c0b7c10 2023-10-30T12:39:45.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:45.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c1d6bf0 con 0x7f1c8c13e730 2023-10-30T12:39:45.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:45.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c91cb4640 10 monclient(hunting): authenticate will time out at 2877.998391s 2023-10-30T12:39:45.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.793+0000 7f1c82ffd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c13e730 0x7f1c8c11f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c8c0b7c10 0x7f1c8c0b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c82ffd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c13e730 0x7f1c8c11f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:53930/0 (socket says 172.21.15.148:53930) 2023-10-30T12:39:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c82ffd640 1 -- 172.21.15.148:0/1274549677 learned_addr learned my addr 172.21.15.148:0/1274549677 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c82ffd640 10 monclient(hunting): get_auth_request con 0x7f1c8c13e730 auth_method 0 2023-10-30T12:39:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:45.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c82ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:45.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c82ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:45.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8bfff640 1 --2- 172.21.15.148:0/1274549677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c8c141940 0x7f1c8c1d7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:45.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8b7fe640 10 monclient(hunting): get_auth_request con 0x7f1c8c0b7c10 auth_method 0 2023-10-30T12:39:45.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8bfff640 10 monclient(hunting): get_auth_request con 0x7f1c8c141940 auth_method 0 2023-10-30T12:39:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:45.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:45.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:45.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:45.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:45.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:45.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.794+0000 7f1c8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:45.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.795+0000 7f1c8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:45.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.795+0000 7f1c8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:45.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.795+0000 7f1c8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:45.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c82ffd640 10 monclient(hunting): handle_auth_done global_id 17696 payload 824 2023-10-30T12:39:45.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c82ffd640 1 -- 172.21.15.148:0/1274549677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c8c0b7c10 msgr2=0x7f1c8c0b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c82ffd640 1 --2- 172.21.15.148:0/1274549677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c8c0b7c10 0x7f1c8c0b0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c82ffd640 1 -- 172.21.15.148:0/1274549677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c8c141940 msgr2=0x7f1c8c1d7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c82ffd640 1 --2- 172.21.15.148:0/1274549677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c8c141940 0x7f1c8c1d7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c82ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:39:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c82ffd640 1 monclient: found mon.noname-c 2023-10-30T12:39:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:45.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c82ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:39:45.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c82ffd640 1 -- 172.21.15.148:0/1274549677 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c11f8f0 con 0x7f1c8c13e730 2023-10-30T12:39:45.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.797+0000 7f1c8b7fe640 1 --2- 172.21.15.148:0/1274549677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c8c0b7c10 0x7f1c8c0b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:39:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.799+0000 7f1c82ffd640 1 --2- 172.21.15.148:0/1274549677 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c13e730 0x7f1c8c11f3b0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009870 tx=0x7f1c740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.799+0000 7f1c88ff9640 10 client.?.objecter ms_handle_connect 0x7f1c8c13e730 2023-10-30T12:39:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.799+0000 7f1c88ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 1 -- 172.21.15.148:0/1274549677 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74032650 con 0x7f1c8c13e730 2023-10-30T12:39:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 10 monclient: dump: 2023-10-30T12:39:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:45.825 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:45.825 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:45.825 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:45.825 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:45.825 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 10 monclient: _finish_auth 0 2023-10-30T12:39:45.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:39:45.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:45.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 1 -- 172.21.15.148:0/1274549677 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7403f030 con 0x7f1c8c13e730 2023-10-30T12:39:45.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c91cb4640 5 monclient: authenticate success, global_id 17696 2023-10-30T12:39:45.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c91cb4640 10 monclient: _renew_subs 2023-10-30T12:39:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c91cb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c91cb4640 1 -- 172.21.15.148:0/1274549677 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c0adf90 con 0x7f1c8c13e730 2023-10-30T12:39:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 1 -- 172.21.15.148:0/1274549677 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74032a00 con 0x7f1c8c13e730 2023-10-30T12:39:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:45.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c88ff9640 10 monclient: dump: 2023-10-30T12:39:45.828 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:45.828 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:45.828 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.828 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:45.829 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:45.829 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:45.829 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:45.829 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:45.829 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:45.830 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:45.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c8a7fc640 4 set_mon_vals no callback set 2023-10-30T12:39:45.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c91cb4640 10 client.17696.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:45.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c91cb4640 10 monclient: _renew_subs 2023-10-30T12:39:45.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c91cb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:45.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c91cb4640 1 -- 172.21.15.148:0/1274549677 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c1d77a0 con 0x7f1c8c13e730 2023-10-30T12:39:45.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.803+0000 7f1c91cb4640 1 librados: init done 2023-10-30T12:39:45.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.804+0000 7f1c91cb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:45.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.804+0000 7f1c91cb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:45.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.804+0000 7f1c91cb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:45.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:45.804+0000 7f1c91cb4640 1 -- 172.21.15.148:0/1274549677 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8c1d69c0 con 0x7f1c8c13e730 2023-10-30T12:39:46.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.049+0000 7f1c88ff9640 1 -- 172.21.15.148:0/1274549677 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f1c74032b70 con 0x7f1c8c13e730 2023-10-30T12:39:46.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.049+0000 7f1c88ff9640 1 --2- 172.21.15.148:0/1274549677 >> v2:172.21.15.148:6800/3205090179 conn(0x7f1c6c06c810 0x7f1c6c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:46.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.049+0000 7f1c8b7fe640 1 --2- 172.21.15.148:0/1274549677 >> v2:172.21.15.148:6800/3205090179 conn(0x7f1c6c06c810 0x7f1c6c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:46.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.050+0000 7f1c8b7fe640 10 monclient: get_auth_request con 0x7f1c6c06c810 auth_method 0 2023-10-30T12:39:46.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.050+0000 7f1c8b7fe640 1 --2- 172.21.15.148:0/1274549677 >> v2:172.21.15.148:6800/3205090179 conn(0x7f1c6c06c810 0x7f1c6c06ecd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c004320 tx=0x7f1c7c007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:46.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.050+0000 7f1c88ff9640 10 client.17696.objecter ms_handle_connect 0x7f1c6c06c810 2023-10-30T12:39:46.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.062+0000 7f1c88ff9640 1 -- 172.21.15.148:0/1274549677 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(899..899 src has 501..899) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f1c74032e20 con 0x7f1c8c13e730 2023-10-30T12:39:46.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.062+0000 7f1c88ff9640 10 client.17696.objecter ms_dispatch 0x7f1c8c000b90 osd_map(899..899 src has 501..899) v4 2023-10-30T12:39:46.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.062+0000 7f1c88ff9640 3 client.17696.objecter handle_osd_map got epochs [899,899] > 0 2023-10-30T12:39:46.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.062+0000 7f1c88ff9640 3 client.17696.objecter handle_osd_map decoding full epoch 899 2023-10-30T12:39:46.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.062+0000 7f1c88ff9640 20 client.17696.objecter dump_active .. 0 homeless 2023-10-30T12:39:46.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.062+0000 7f1c88ff9640 1 -- 172.21.15.148:0/1274549677 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1c74037070 con 0x7f1c8c13e730 2023-10-30T12:39:46.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.062+0000 7f1c88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:46.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.062+0000 7f1c88ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.232+0000 7f1c91cb4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:39:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.232+0000 7f1c91cb4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:39:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.232+0000 7f1c91cb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.232+0000 7f1c91cb4640 1 -- 172.21.15.148:0/1274549677 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c8c0aa6a0 con 0x7f1c8c13e730 2023-10-30T12:39:46.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.276+0000 7f1c88ff9640 1 -- 172.21.15.148:0/1274549677 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v899) v1 ==== 74+0+15586 (secure 0 0 0) 0x7f1c74079740 con 0x7f1c8c13e730 2023-10-30T12:39:46.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.276+0000 7f1c88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:39:46.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.276+0000 7f1c88ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:39:46.279 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:39:46.279 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":899,"fsid":"e2fdab2f-1adf-4b35-9eee-e40114aa8bbb","created":"2023-10-30T12:17:03.292319+0000","modified":"2023-10-30T12:39:43.355351+0000","last_up_change":"2023-10-30T12:17:51.437771+0000","last_in_change":"2023-10-30T12:17:09.070671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":103,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:17:53.315604+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T12:17:58.245447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-10-30T12:19:16.803884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"183f5799-ff32-4ada-b4ff-fd8fcbd82d08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":2395694124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2395694124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2395694124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2395694124}]},"public_addr":"172.21.15.148:6805/2395694124","cluster_addr":"172.21.15.148:6807/2395694124","heartbeat_back_addr":"172.21.15.148:6810/2395694124","heartbeat_front_addr":"172.21.15.148:6808/2395694124","state":["exists","up"]},{"osd":1,"uuid":"ee28bf60-9f29-4fb1-87a0-7e0707b143b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":912412250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":912412250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":912412250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":912412250}]},"public_addr":"172.21.15.148:6813/912412250","cluster_addr":"172.21.15.148:6814/912412250","heartbeat_back_addr":"172.21.15.148:6816/912412250","heartbeat_front_addr":"172.21.15.148:6815/912412250","state":["exists","up"]},{"osd":2,"uuid":"22573689-d164-4530-aa70-adbab2232e1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":2368806934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2368806934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":2368806934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":2368806934}]},"public_addr":"172.21.15.148:6801/2368806934","cluster_addr":"172.21.15.148:6802/2368806934","heartbeat_back_addr":"172.21.15.148:6804/2368806934","heartbeat_front_addr":"172.21.15.148:6803/2368806934","state":["exists","up"]},{"osd":3,"uuid":"36c6a615-3a6a-4c78-86ed-546ddf04aa1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":2233349848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":2233349848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2233349848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":2233349848}]},"public_addr":"172.21.15.148:6806/2233349848","cluster_addr":"172.21.15.148:6809/2233349848","heartbeat_back_addr":"172.21.15.148:6812/2233349848","heartbeat_front_addr":"172.21.15.148:6811/2233349848","state":["exists","up"]},{"osd":4,"uuid":"2cb5add2-1200-41f1-8e70-78abdc2df024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":887,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2836839739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":2836839739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":2836839739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2836839739}]},"public_addr":"172.21.15.150:6808/2836839739","cluster_addr":"172.21.15.150:6809/2836839739","heartbeat_back_addr":"172.21.15.150:6811/2836839739","heartbeat_front_addr":"172.21.15.150:6810/2836839739","state":["exists","up"]},{"osd":5,"uuid":"a2ab2b14-67a0-4d44-9396-d7a7d6cdf1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4190376848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":4190376848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6807","nonce":4190376848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4190376848}]},"public_addr":"172.21.15.150:6804/4190376848","cluster_addr":"172.21.15.150:6805/4190376848","heartbeat_back_addr":"172.21.15.150:6807/4190376848","heartbeat_front_addr":"172.21.15.150:6806/4190376848","state":["exists","up"]},{"osd":6,"uuid":"150704c8-4d6c-4df9-a728-25683a64db84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2604225934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6801","nonce":2604225934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":2604225934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2604225934}]},"public_addr":"172.21.15.150:6800/2604225934","cluster_addr":"172.21.15.150:6801/2604225934","heartbeat_back_addr":"172.21.15.150:6803/2604225934","heartbeat_front_addr":"172.21.15.150:6802/2604225934","state":["exists","up"]},{"osd":7,"uuid":"5daaf214-6591-4913-a103-5eb53ad2937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3463178370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":3463178370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":3463178370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3463178370}]},"public_addr":"172.21.15.150:6812/3463178370","cluster_addr":"172.21.15.150:6813/3463178370","heartbeat_back_addr":"172.21.15.150:6815/3463178370","heartbeat_front_addr":"172.21.15.150:6814/3463178370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.920964+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.821114+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.365893+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:49.037497+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.184019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.644370+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:47.688638+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:17:48.429422+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":9,"length":2}]},{"pool":10,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":60,"snaps":[{"begin":6,"length":2},{"begin":11,"length":2}]},{"pool":86,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":97,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":98,"snaps":[{"begin":2,"length":1}]},{"pool":109,"snaps":[{"begin":4,"length":4}]},{"pool":110,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":120,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":173,"snaps":[{"begin":12,"length":1}]},{"pool":184,"snaps":[{"begin":14,"length":1}]},{"pool":189,"snaps":[{"begin":17,"length":1}]},{"pool":192,"snaps":[{"begin":19,"length":2}]},{"pool":197,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:39:46.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 10 librados: watch_flush enter 2023-10-30T12:39:46.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 10 librados: watch_flush exit 2023-10-30T12:39:46.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 20 client.17696.objecter shutdown clearing up homeless session... 2023-10-30T12:39:46.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 10 client.17696.objecter successfully canceled tick 2023-10-30T12:39:46.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 -- 172.21.15.148:0/1274549677 >> v2:172.21.15.148:6800/3205090179 conn(0x7f1c6c06c810 msgr2=0x7f1c6c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:46.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 --2- 172.21.15.148:0/1274549677 >> v2:172.21.15.148:6800/3205090179 conn(0x7f1c6c06c810 0x7f1c6c06ecd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c004320 tx=0x7f1c7c007e30 comp rx=0 tx=0).stop 2023-10-30T12:39:46.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 10 monclient: shutdown 2023-10-30T12:39:46.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:46.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 -- 172.21.15.148:0/1274549677 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c13e730 msgr2=0x7f1c8c11f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:46.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 --2- 172.21.15.148:0/1274549677 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c13e730 0x7f1c8c11f3b0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009870 tx=0x7f1c740046c0 comp rx=0 tx=0).stop 2023-10-30T12:39:46.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 -- 172.21.15.148:0/1274549677 shutdown_connections 2023-10-30T12:39:46.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 --2- 172.21.15.148:0/1274549677 >> v2:172.21.15.148:6800/3205090179 conn(0x7f1c6c06c810 0x7f1c6c06ecd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:46.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 --2- 172.21.15.148:0/1274549677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c8c141940 0x7f1c8c1d7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:46.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 --2- 172.21.15.148:0/1274549677 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1c8c13e730 0x7f1c8c11f3b0 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:46.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 --2- 172.21.15.148:0/1274549677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c8c0b7c10 0x7f1c8c0b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:46.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 -- 172.21.15.148:0/1274549677 >> 172.21.15.148:0/1274549677 conn(0x7f1c8c06c660 msgr2=0x7f1c8c13c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:46.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 -- 172.21.15.148:0/1274549677 shutdown_connections 2023-10-30T12:39:46.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.278+0000 7f1c91cb4640 1 -- 172.21.15.148:0/1274549677 wait complete. 2023-10-30T12:39:46.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:46.279+0000 7f1c91cb4640 1 librados: shutdown 2023-10-30T12:39:47.292 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T12:39:47.293 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:39:47.357 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:02.431 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:47.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.400+0000 7f0e424cf640 10 monclient: get_monmap_and_config 2023-10-30T12:39:47.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.400+0000 7f0e424cf640 10 monclient: build_initial_monmap 2023-10-30T12:39:47.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.400+0000 7f0e424cf640 10 monclient: monmap: 2023-10-30T12:39:47.403 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:47.403 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:47.405 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:47.401851+0000 2023-10-30T12:39:47.405 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:47.401851+0000 2023-10-30T12:39:47.406 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:47.406 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:47.406 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:47.406 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:47.406 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:47.407 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:47.407 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:47.407 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.408 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:02.451 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:47.408 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:02.454 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:47.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.401+0000 7f0e424cf640 1 Processor -- start 2023-10-30T12:39:47.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.401+0000 7f0e424cf640 1 -- start start 2023-10-30T12:39:47.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.401+0000 7f0e424cf640 10 monclient: init 2023-10-30T12:39:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-10-30T12:39:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b7c10 0x7f0e3c0a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 10 monclient(hunting): picked mon.noname-b con 0x7f0e3c0b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:47.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0e3c141940 0x7f0e3c141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 10 monclient(hunting): picked mon.noname-c con 0x7f0e3c141940 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:47.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e3c0b1c10 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:47.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3c0a9350 con 0x7f0e3c0b1c10 2023-10-30T12:39:47.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:47.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e3c0670a0 con 0x7f0e3c141940 2023-10-30T12:39:47.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:47.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c0b7ff0 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:47.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e424cf640 10 monclient(hunting): authenticate will time out at 2879.607122s 2023-10-30T12:39:47.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.402+0000 7f0e3bfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b7c10 0x7f0e3c0a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:47.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.403+0000 7f0e3bfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b7c10 0x7f0e3c0a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:53940/0 (socket says 172.21.15.148:53940) 2023-10-30T12:39:47.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.403+0000 7f0e3bfff640 1 -- 172.21.15.148:0/4257360898 learned_addr learned my addr 172.21.15.148:0/4257360898 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:47.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.403+0000 7f0e3bfff640 10 monclient(hunting): get_auth_request con 0x7f0e3c0b7c10 auth_method 0 2023-10-30T12:39:47.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.403+0000 7f0e3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:47.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.403+0000 7f0e3bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:47.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.403+0000 7f0e3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:47.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.403+0000 7f0e3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:47.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.403+0000 7f0e3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:47.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.403+0000 7f0e3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:47.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.405+0000 7f0e3bfff640 10 monclient(hunting): handle_auth_done global_id 17699 payload 274 2023-10-30T12:39:47.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.405+0000 7f0e3bfff640 1 -- 172.21.15.148:0/4257360898 >> v2:172.21.15.150:3300/0 conn(0x7f0e3c141940 msgr2=0x7f0e3c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:47.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.405+0000 7f0e3bfff640 1 --2- 172.21.15.148:0/4257360898 >> v2:172.21.15.150:3300/0 conn(0x7f0e3c141940 0x7f0e3c141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.405+0000 7f0e3bfff640 1 -- 172.21.15.148:0/4257360898 >> v1:172.21.15.148:6789/0 conn(0x7f0e3c0b1c10 legacy=0x7f0e3c0b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:47.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.405+0000 7f0e3bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:39:47.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.405+0000 7f0e3bfff640 1 monclient: found mon.noname-b 2023-10-30T12:39:47.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.405+0000 7f0e3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:47.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.405+0000 7f0e3bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:47.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.405+0000 7f0e3bfff640 1 -- 172.21.15.148:0/4257360898 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c0aa6a0 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.417 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:02.482 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:47.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.409+0000 7f0e3bfff640 1 --2- 172.21.15.148:0/4257360898 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b7c10 0x7f0e3c0a8aa0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009870 tx=0x7f0e3002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f58eae82fe952ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:47.428 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:02.476 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:47.430 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:02.504 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:47.462 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:02.535 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:47.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 1 -- 172.21.15.148:0/4257360898 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e300317c0 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:47.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:47.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 10 monclient: dump: 2023-10-30T12:39:47.466 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:47.467 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:47.467 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.467 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.467 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:47.467 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:47.468 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:47.468 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:47.468 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:47.468 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 10 monclient: _finish_auth 0 2023-10-30T12:39:47.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 10 monclient: _check_auth_tickets 2023-10-30T12:39:47.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:47.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 1 -- 172.21.15.148:0/4257360898 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3002e070 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e424cf640 5 monclient: authenticate success, global_id 17699 2023-10-30T12:39:47.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:47.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e424cf640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:47.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e424cf640 4 set_mon_vals no callback set 2023-10-30T12:39:47.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 1 -- 172.21.15.148:0/4257360898 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e30031b50 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:47.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:47.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e3affd640 10 monclient: dump: 2023-10-30T12:39:47.471 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:47.471 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:47.471 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:47.473 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e424cf640 10 monclient: shutdown 2023-10-30T12:39:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e424cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e424cf640 1 -- 172.21.15.148:0/4257360898 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b7c10 msgr2=0x7f0e3c0a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.464+0000 7f0e424cf640 1 --2- 172.21.15.148:0/4257360898 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b7c10 0x7f0e3c0a8aa0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009870 tx=0x7f0e3002b860 comp rx=0 tx=0).stop 2023-10-30T12:39:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.465+0000 7f0e424cf640 1 -- 172.21.15.148:0/4257360898 shutdown_connections 2023-10-30T12:39:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.465+0000 7f0e424cf640 1 --2- 172.21.15.148:0/4257360898 >> v2:172.21.15.150:3300/0 conn(0x7f0e3c141940 0x7f0e3c141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.465+0000 7f0e424cf640 1 --2- 172.21.15.148:0/4257360898 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b7c10 0x7f0e3c0a8aa0 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.465+0000 7f0e424cf640 1 -- 172.21.15.148:0/4257360898 >> 172.21.15.148:0/4257360898 conn(0x7f0e3c06c660 msgr2=0x7f0e3c0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.465+0000 7f0e424cf640 1 -- 172.21.15.148:0/4257360898 shutdown_connections 2023-10-30T12:39:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.465+0000 7f0e424cf640 1 -- 172.21.15.148:0/4257360898 wait complete. 2023-10-30T12:39:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.466+0000 7f0e424cf640 10 monclient: build_initial_monmap 2023-10-30T12:39:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.466+0000 7f0e424cf640 10 monclient: monmap: 2023-10-30T12:39:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:47.478 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.478 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:02.541 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.466+0000 7f0e424cf640 1 librados: starting msgr at 2023-10-30T12:39:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.466+0000 7f0e424cf640 1 librados: starting objecter 2023-10-30T12:39:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.466+0000 7f0e424cf640 1 Processor -- start 2023-10-30T12:39:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.466+0000 7f0e424cf640 1 -- start start 2023-10-30T12:39:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.466+0000 7f0e424cf640 1 librados: setting wanted keys 2023-10-30T12:39:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.466+0000 7f0e424cf640 1 librados: calling monclient init 2023-10-30T12:39:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.466+0000 7f0e424cf640 10 monclient: init 2023-10-30T12:39:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:39:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b1c10 0x7f0e3c0ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 10 monclient(hunting): picked mon.noname-c con 0x7f0e3c0b1c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e3c0b7c10 0x7f0e3c0b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 10 monclient(hunting): picked mon.noname-a con 0x7f0e3c0b7c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e3c141940 0x7f0e3c11f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 10 monclient(hunting): picked mon.noname-b con 0x7f0e3c141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c0a84d0 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c1d9ee0 con 0x7f0e3c141940 2023-10-30T12:39:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c11fbd0 con 0x7f0e3c0b1c10 2023-10-30T12:39:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.467+0000 7f0e424cf640 10 monclient(hunting): authenticate will time out at 2879.672745s 2023-10-30T12:39:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.470+0000 7f0e3b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e3c0b7c10 0x7f0e3c0b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.474+0000 7f0e3b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e3c0b7c10 0x7f0e3c0b0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55184/0 (socket says 172.21.15.148:55184) 2023-10-30T12:39:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.474+0000 7f0e3b7fe640 1 -- 172.21.15.148:0/4223160015 learned_addr learned my addr 172.21.15.148:0/4223160015 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.474+0000 7f0e3b7fe640 10 monclient(hunting): get_auth_request con 0x7f0e3c0b7c10 auth_method 0 2023-10-30T12:39:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.474+0000 7f0e3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.474+0000 7f0e3b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.474+0000 7f0e3b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.478+0000 7f0e3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.478+0000 7f0e3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.478+0000 7f0e3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.480+0000 7f0e3bfff640 1 --2- 172.21.15.148:0/4223160015 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b1c10 0x7f0e3c0ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.483+0000 7f0e3bfff640 10 monclient(hunting): get_auth_request con 0x7f0e3c0b1c10 auth_method 0 2023-10-30T12:39:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.483+0000 7f0e3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.483+0000 7f0e3bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.483+0000 7f0e3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3b7fe640 10 monclient(hunting): handle_auth_done global_id 16248 payload 824 2023-10-30T12:39:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3b7fe640 1 -- 172.21.15.148:0/4223160015 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b1c10 msgr2=0x7f0e3c0ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3b7fe640 1 --2- 172.21.15.148:0/4223160015 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0e3c0b1c10 0x7f0e3c0ae1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3b7fe640 1 -- 172.21.15.148:0/4223160015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e3c141940 msgr2=0x7f0e3c11f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3b7fe640 1 --2- 172.21.15.148:0/4223160015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e3c141940 0x7f0e3c11f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:39:47.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:39:47.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:47.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:39:47.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.485+0000 7f0e3b7fe640 1 -- 172.21.15.148:0/4223160015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c1da650 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.491+0000 7f0e3b7fe640 1 --2- 172.21.15.148:0/4223160015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e3c0b7c10 0x7f0e3c0b0d30 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00b410 tx=0x7f0e2c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.491+0000 7f0e38ff9640 10 client.?.objecter ms_handle_connect 0x7f0e3c0b7c10 2023-10-30T12:39:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.491+0000 7f0e38ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:47.496 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:02.570 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:47.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e38ff9640 1 -- 172.21.15.148:0/4223160015 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c011830 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:47.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e38ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:39:47.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e38ff9640 10 monclient: dump: 2023-10-30T12:39:47.504 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:47.504 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:47.504 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.505 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.505 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:47.505 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:47.505 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:47.505 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:47.506 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:47.506 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e38ff9640 10 monclient: _finish_auth 0 2023-10-30T12:39:47.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e38ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:39:47.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:47.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e38ff9640 1 -- 172.21.15.148:0/4223160015 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c00d040 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e424cf640 5 monclient: authenticate success, global_id 16248 2023-10-30T12:39:47.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e424cf640 10 monclient: _renew_subs 2023-10-30T12:39:47.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e424cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:39:47.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3c1da8b0 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e38ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.500+0000 7f0e38ff9640 1 -- 172.21.15.148:0/4223160015 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c011c00 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:39:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e38ff9640 10 monclient: dump: 2023-10-30T12:39:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:47.509 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:47.509 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.509 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.509 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:47.509 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:47.510 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:47.510 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:47.510 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:47.510 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e3a7fc640 4 set_mon_vals no callback set 2023-10-30T12:39:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 10 client.16248.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 10 monclient: _renew_subs 2023-10-30T12:39:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:39:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3c1dadd0 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 1 librados: init done 2023-10-30T12:39:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 10 client.16248.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:47.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 15 client.16248.objecter _session_command_op_assign -1 1 2023-10-30T12:39:47.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 20 client.16248.objecter _get_session osd=-1 returning homeless 2023-10-30T12:39:47.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 10 client.16248.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:47.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:39:47.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.501+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e3c0a8fe0 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.532+0000 7f0e38ff9640 1 -- 172.21.15.148:0/4223160015 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f0e2c01a400 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.533+0000 7f0e38ff9640 1 --2- 172.21.15.148:0/4223160015 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0e2806c810 0x7f0e2806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.533+0000 7f0e3bfff640 1 --2- 172.21.15.148:0/4223160015 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0e2806c810 0x7f0e2806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:47.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.533+0000 7f0e3bfff640 10 monclient: get_auth_request con 0x7f0e2806c810 auth_method 0 2023-10-30T12:39:47.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.533+0000 7f0e38ff9640 10 client.16248.objecter ms_handle_connect 0x7f0e2806c810 2023-10-30T12:39:47.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.533+0000 7f0e3bfff640 1 --2- 172.21.15.148:0/4223160015 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0e2806c810 0x7f0e2806ecd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0e30036000 tx=0x7f0e3003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:47.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 1 -- 172.21.15.148:0/4223160015 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f0e2c058c70 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 10 client.16248.objecter ms_dispatch 0x7f0e3c000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:47.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 3 client.16248.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:47.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 3 client.16248.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:47.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 10 client.16248.objecter checking command 1 2023-10-30T12:39:47.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 1 --2- 172.21.15.148:0/4223160015 >> v2:172.21.15.148:6805/2395694124 conn(0x7f0e28073b30 0x7f0e28075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 20 client.16248.objecter _get_session s=0x7f0e2c011c00 osd=0 3 2023-10-30T12:39:47.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 20 client.16248.objecter put_session s=0x7f0e2c011c00 osd=0 3 2023-10-30T12:39:47.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 15 client.16248.objecter _session_command_op_remove -1 1 2023-10-30T12:39:47.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 20 client.16248.objecter _get_session s=0x7f0e2c011c00 osd=0 3 2023-10-30T12:39:47.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 20 client.16248.objecter get_session s=0x7f0e2c011c00 osd=0 3 2023-10-30T12:39:47.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 15 client.16248.objecter _session_command_op_assign 0 1 2023-10-30T12:39:47.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 20 client.16248.objecter put_session s=0x7f0e2c011c00 osd=0 4 2023-10-30T12:39:47.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 10 client.16248.objecter _send_command 1 2023-10-30T12:39:47.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 1 -- 172.21.15.148:0/4223160015 --> v2:172.21.15.148:6805/2395694124 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e2c011830 con 0x7f0e28073b30 2023-10-30T12:39:47.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 20 client.16248.objecter dump_active .. 0 homeless 2023-10-30T12:39:47.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 1 -- 172.21.15.148:0/4223160015 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e2c009230 con 0x7f0e3c0b7c10 2023-10-30T12:39:47.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.569+0000 7f0e38ff9640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:39:47.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.570+0000 7f0e40a6c640 1 --2- 172.21.15.148:0/4223160015 >> v2:172.21.15.148:6805/2395694124 conn(0x7f0e28073b30 0x7f0e28075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:47.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.570+0000 7f0e40a6c640 10 monclient: get_auth_request con 0x7f0e28073b30 auth_method 0 2023-10-30T12:39:47.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.572+0000 7f0e38ff9640 10 client.16248.objecter ms_handle_connect 0x7f0e28073b30 2023-10-30T12:39:47.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.572+0000 7f0e40a6c640 1 --2- 172.21.15.148:0/4223160015 >> v2:172.21.15.148:6805/2395694124 conn(0x7f0e28073b30 0x7f0e28075fd0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:47.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.612+0000 7f0e38ff9640 1 -- 172.21.15.148:0/4223160015 <== osd.0 v2:172.21.15.148:6805/2395694124 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0e2c011830 con 0x7f0e28073b30 2023-10-30T12:39:47.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.612+0000 7f0e38ff9640 10 client.16248.objecter ms_dispatch 0x7f0e3c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:39:47.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.612+0000 7f0e38ff9640 10 client.16248.objecter _finish_command 1 = system:0 2023-10-30T12:39:47.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.612+0000 7f0e38ff9640 20 client.16248.objecter put_session s=0x7f0e2c011c00 osd=0 4 2023-10-30T12:39:47.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.612+0000 7f0e38ff9640 15 client.16248.objecter _session_command_op_remove 0 1 2023-10-30T12:39:47.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 10 client.16248.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:39:47.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 15 client.16248.objecter _session_command_op_assign -1 2 2023-10-30T12:39:47.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 20 client.16248.objecter _get_session s=0x7f0e2c011c00 osd=0 3 2023-10-30T12:39:47.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 20 client.16248.objecter put_session s=0x7f0e2c011c00 osd=0 3 2023-10-30T12:39:47.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 20 client.16248.objecter _get_session s=0x7f0e2c011c00 osd=0 3 2023-10-30T12:39:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 15 client.16248.objecter _session_command_op_remove -1 2 2023-10-30T12:39:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 20 client.16248.objecter get_session s=0x7f0e2c011c00 osd=0 3 2023-10-30T12:39:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 15 client.16248.objecter _session_command_op_assign 0 2 2023-10-30T12:39:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 20 client.16248.objecter put_session s=0x7f0e2c011c00 osd=0 4 2023-10-30T12:39:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 10 client.16248.objecter _send_command 2 2023-10-30T12:39:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.631+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 --> v2:172.21.15.148:6805/2395694124 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0e3c1422a0 con 0x7f0e28073b30 2023-10-30T12:39:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e38ff9640 1 -- 172.21.15.148:0/4223160015 <== osd.0 v2:172.21.15.148:6805/2395694124 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0e34006b10 con 0x7f0e28073b30 2023-10-30T12:39:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e38ff9640 10 client.16248.objecter ms_dispatch 0x7f0e3c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:39:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e38ff9640 10 client.16248.objecter _finish_command 2 = system:0 2023-10-30T12:39:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e38ff9640 20 client.16248.objecter put_session s=0x7f0e2c011c00 osd=0 4 2023-10-30T12:39:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e38ff9640 15 client.16248.objecter _session_command_op_remove 0 2 2023-10-30T12:39:47.649 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T12:39:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:39:47.649 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T12:39:47.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 10 librados: watch_flush enter 2023-10-30T12:39:47.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 10 librados: watch_flush exit 2023-10-30T12:39:47.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 10 client.16248.objecter close_session for osd.0 2023-10-30T12:39:47.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 >> v2:172.21.15.148:6805/2395694124 conn(0x7f0e28073b30 msgr2=0x7f0e28075fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:47.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 1 --2- 172.21.15.148:0/4223160015 >> v2:172.21.15.148:6805/2395694124 conn(0x7f0e28073b30 0x7f0e28075fd0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 20 client.16248.objecter put_session s=0x7f0e2c011c00 osd=0 1 2023-10-30T12:39:47.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 20 client.16248.objecter shutdown clearing up homeless session... 2023-10-30T12:39:47.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 10 client.16248.objecter successfully canceled tick 2023-10-30T12:39:47.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0e2806c810 msgr2=0x7f0e2806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:47.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 1 --2- 172.21.15.148:0/4223160015 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0e2806c810 0x7f0e2806ecd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0e30036000 tx=0x7f0e3003d040 comp rx=0 tx=0).stop 2023-10-30T12:39:47.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 10 monclient: shutdown 2023-10-30T12:39:47.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:47.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e3c0b7c10 msgr2=0x7f0e3c0b0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:47.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 1 --2- 172.21.15.148:0/4223160015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e3c0b7c10 0x7f0e3c0b0d30 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00b410 tx=0x7f0e2c00b8d0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e40a6c640 1 -- 172.21.15.148:0/4223160015 reap_dead start 2023-10-30T12:39:47.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 shutdown_connections 2023-10-30T12:39:47.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 >> 172.21.15.148:0/4223160015 conn(0x7f0e3c06c660 msgr2=0x7f0e3c13f840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:47.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.646+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 shutdown_connections 2023-10-30T12:39:47.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.647+0000 7f0e424cf640 1 -- 172.21.15.148:0/4223160015 wait complete. 2023-10-30T12:39:47.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.647+0000 7f0e424cf640 1 librados: shutdown 2023-10-30T12:39:47.659 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T12:39:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.778+0000 7f20653ce640 10 monclient: get_monmap_and_config 2023-10-30T12:39:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.778+0000 7f20653ce640 10 monclient: build_initial_monmap 2023-10-30T12:39:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.778+0000 7f20653ce640 10 monclient: monmap: 2023-10-30T12:39:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:47.779738+0000 2023-10-30T12:39:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:47.779738+0000 2023-10-30T12:39:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:47.783 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.779+0000 7f20653ce640 1 Processor -- start 2023-10-30T12:39:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.779+0000 7f20653ce640 1 -- start start 2023-10-30T12:39:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.779+0000 7f20653ce640 10 monclient: init 2023-10-30T12:39:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-10-30T12:39:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f206006a0e0 0x7f20600aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 10 monclient(hunting): picked mon.noname-c con 0x7f206006a0e0 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f206013ed20 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2060141ba0 0x7f2060141fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 10 monclient(hunting): picked mon.noname-b con 0x7f2060141ba0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20600aeb80 con 0x7f206013ed20 2023-10-30T12:39:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2060067340 con 0x7f206006a0e0 2023-10-30T12:39:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206006a4c0 con 0x7f2060141ba0 2023-10-30T12:39:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.780+0000 7f20653ce640 10 monclient(hunting): authenticate will time out at 2879.985088s 2023-10-30T12:39:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.784+0000 7f205effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f206006a0e0 0x7f20600aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:47.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.793+0000 7f205f7fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2060141ba0 0x7f2060141fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:47.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.796+0000 7f205f7fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2060141ba0 0x7f2060141fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:53958/0 (socket says 172.21.15.148:53958) 2023-10-30T12:39:47.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.796+0000 7f205f7fe640 1 -- 172.21.15.148:0/821485330 learned_addr learned my addr 172.21.15.148:0/821485330 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:47.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.796+0000 7f205f7fe640 10 monclient(hunting): get_auth_request con 0x7f2060141ba0 auth_method 0 2023-10-30T12:39:47.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.796+0000 7f205f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:47.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.796+0000 7f205f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:47.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.796+0000 7f205f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:47.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.800+0000 7f205effd640 10 monclient(hunting): get_auth_request con 0x7f206006a0e0 auth_method 0 2023-10-30T12:39:47.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.800+0000 7f205effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:47.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.800+0000 7f205effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:47.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.800+0000 7f205effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:47.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.800+0000 7f205effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:47.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.800+0000 7f205effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:47.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.800+0000 7f205effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:47.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.801+0000 7f205f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:47.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.801+0000 7f205f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:47.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.801+0000 7f205f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:47.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.802+0000 7f205effd640 10 monclient(hunting): handle_auth_done global_id 22066 payload 274 2023-10-30T12:39:47.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.802+0000 7f205effd640 1 -- 172.21.15.148:0/821485330 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2060141ba0 msgr2=0x7f2060141fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:47.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.802+0000 7f205effd640 1 --2- 172.21.15.148:0/821485330 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2060141ba0 0x7f2060141fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.802+0000 7f205effd640 1 -- 172.21.15.148:0/821485330 >> v1:172.21.15.150:6789/0 conn(0x7f206013ed20 legacy=0x7f206013f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:47.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.802+0000 7f205effd640 10 monclient: _finish_hunting 0 2023-10-30T12:39:47.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.802+0000 7f205effd640 1 monclient: found mon.noname-c 2023-10-30T12:39:47.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.802+0000 7f205effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:47.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.802+0000 7f205effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:39:47.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.802+0000 7f205effd640 1 -- 172.21.15.148:0/821485330 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20600af7a0 con 0x7f206006a0e0 2023-10-30T12:39:47.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.808+0000 7f205effd640 1 --2- 172.21.15.148:0/821485330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f206006a0e0 0x7f20600aef40 secure :-1 s=READY pgs=5435 cs=0 l=1 rev1=1 crypto rx=0x7f206006a610 tx=0x7f205002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f539846551886da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:47.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.817+0000 7f205e7fc640 1 -- 172.21.15.148:0/821485330 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050031800 con 0x7f206006a0e0 2023-10-30T12:39:47.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.817+0000 7f205e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:47.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.817+0000 7f205e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:47.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.817+0000 7f205e7fc640 10 monclient: dump: 2023-10-30T12:39:47.820 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:47.820 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:47.820 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.821 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.821 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:47.821 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:47.821 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:47.822 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:47.822 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:47.822 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.817+0000 7f205e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:39:47.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.817+0000 7f205e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:39:47.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.817+0000 7f205e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:47.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f205e7fc640 1 -- 172.21.15.148:0/821485330 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2050031970 con 0x7f206006a0e0 2023-10-30T12:39:47.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f205e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:47.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f205e7fc640 1 -- 172.21.15.148:0/821485330 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050031d10 con 0x7f206006a0e0 2023-10-30T12:39:47.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 5 monclient: authenticate success, global_id 22066 2023-10-30T12:39:47.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:47.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 4 set_mon_vals no callback set 2023-10-30T12:39:47.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f205e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:47.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f205e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:47.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f205e7fc640 10 monclient: dump: 2023-10-30T12:39:47.825 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:47.825 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:47.825 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.825 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.825 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:47.826 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:47.826 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:47.826 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:47.826 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:47.826 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 10 monclient: shutdown 2023-10-30T12:39:47.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:47.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 1 -- 172.21.15.148:0/821485330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f206006a0e0 msgr2=0x7f20600aef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:47.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 1 --2- 172.21.15.148:0/821485330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f206006a0e0 0x7f20600aef40 secure :-1 s=READY pgs=5435 cs=0 l=1 rev1=1 crypto rx=0x7f206006a610 tx=0x7f205002b8a0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 1 -- 172.21.15.148:0/821485330 shutdown_connections 2023-10-30T12:39:47.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 1 --2- 172.21.15.148:0/821485330 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2060141ba0 0x7f2060141fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 1 --2- 172.21.15.148:0/821485330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f206006a0e0 0x7f20600aef40 unknown :-1 s=CLOSED pgs=5435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 1 -- 172.21.15.148:0/821485330 >> 172.21.15.148:0/821485330 conn(0x7f206006b0b0 msgr2=0x7f206006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:47.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 1 -- 172.21.15.148:0/821485330 shutdown_connections 2023-10-30T12:39:47.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.818+0000 7f20653ce640 1 -- 172.21.15.148:0/821485330 wait complete. 2023-10-30T12:39:47.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.819+0000 7f20653ce640 10 monclient: build_initial_monmap 2023-10-30T12:39:47.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.819+0000 7f20653ce640 10 monclient: monmap: 2023-10-30T12:39:47.832 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:47.832 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:47.832 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:47.832 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:47.833 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:47.833 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:47.833 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:47.833 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:47.833 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:47.833 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.819+0000 7f20653ce640 1 librados: starting msgr at 2023-10-30T12:39:47.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.819+0000 7f20653ce640 1 librados: starting objecter 2023-10-30T12:39:47.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.819+0000 7f20653ce640 1 Processor -- start 2023-10-30T12:39:47.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.819+0000 7f20653ce640 1 -- start start 2023-10-30T12:39:47.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.819+0000 7f20653ce640 1 librados: setting wanted keys 2023-10-30T12:39:47.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.819+0000 7f20653ce640 1 librados: calling monclient init 2023-10-30T12:39:47.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.819+0000 7f20653ce640 10 monclient: init 2023-10-30T12:39:47.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:47.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:39:47.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f206006a0e0 0x7f206006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 10 monclient(hunting): picked mon.noname-c con 0x7f206006a0e0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:47.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f206013ed20 0x7f20600701f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 10 monclient(hunting): picked mon.noname-a con 0x7f206013ed20 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2060141ba0 0x7f20600b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 10 monclient(hunting): picked mon.noname-b con 0x7f2060141ba0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20600ad9a0 con 0x7f206013ed20 2023-10-30T12:39:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:47.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2060070950 con 0x7f2060141ba0 2023-10-30T12:39:47.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:47.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2060071920 con 0x7f206006a0e0 2023-10-30T12:39:47.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:47.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.820+0000 7f20653ce640 10 monclient(hunting): authenticate will time out at 2880.025521s 2023-10-30T12:39:47.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.822+0000 7f205f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2060141ba0 0x7f20600b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:47.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.823+0000 7f2057fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f206013ed20 0x7f20600701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:47.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.826+0000 7f205f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2060141ba0 0x7f20600b2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:55540/0 (socket says 172.21.15.148:55540) 2023-10-30T12:39:47.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.826+0000 7f205f7fe640 1 -- 172.21.15.148:0/400134679 learned_addr learned my addr 172.21.15.148:0/400134679 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:47.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.826+0000 7f205f7fe640 10 monclient(hunting): get_auth_request con 0x7f2060141ba0 auth_method 0 2023-10-30T12:39:47.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.826+0000 7f205f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:47.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.826+0000 7f205f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:47.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.826+0000 7f205f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:47.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.826+0000 7f2057fff640 10 monclient(hunting): get_auth_request con 0x7f206013ed20 auth_method 0 2023-10-30T12:39:47.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.826+0000 7f2057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:47.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.826+0000 7f2057fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:47.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.826+0000 7f2057fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:47.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.828+0000 7f205f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:47.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.828+0000 7f205f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:47.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.828+0000 7f205f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:47.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.830+0000 7f205effd640 1 --2- 172.21.15.148:0/400134679 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f206006a0e0 0x7f206006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:47.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.832+0000 7f205f7fe640 10 monclient(hunting): handle_auth_done global_id 22069 payload 824 2023-10-30T12:39:47.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.832+0000 7f205f7fe640 1 -- 172.21.15.148:0/400134679 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f206006a0e0 msgr2=0x7f206006faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:47.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.832+0000 7f205f7fe640 1 --2- 172.21.15.148:0/400134679 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f206006a0e0 0x7f206006faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.832+0000 7f205f7fe640 1 -- 172.21.15.148:0/400134679 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f206013ed20 msgr2=0x7f20600701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:47.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.832+0000 7f205f7fe640 1 --2- 172.21.15.148:0/400134679 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f206013ed20 0x7f20600701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:47.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.832+0000 7f205f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:39:47.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.832+0000 7f205f7fe640 1 monclient: found mon.noname-b 2023-10-30T12:39:47.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.832+0000 7f205f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:47.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.832+0000 7f205f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:39:47.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.832+0000 7f205f7fe640 1 -- 172.21.15.148:0/400134679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2060070de0 con 0x7f2060141ba0 2023-10-30T12:39:47.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.835+0000 7f205f7fe640 1 --2- 172.21.15.148:0/400134679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2060141ba0 0x7f20600b2430 secure :-1 s=READY pgs=5436 cs=0 l=1 rev1=1 crypto rx=0x7f205800ec10 tx=0x7f205800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:47.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.835+0000 7f20577fe640 10 client.?.objecter ms_handle_connect 0x7f2060141ba0 2023-10-30T12:39:47.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.835+0000 7f20577fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:47.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 1 -- 172.21.15.148:0/400134679 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205800f040 con 0x7f2060141ba0 2023-10-30T12:39:47.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:47.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:47.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 10 monclient: dump: 2023-10-30T12:39:47.847 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:47.847 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:47.848 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.848 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.848 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:47.848 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:47.848 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:47.849 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:47.849 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:47.849 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 10 monclient: _finish_auth 0 2023-10-30T12:39:47.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 10 monclient: _check_auth_tickets 2023-10-30T12:39:47.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:47.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 1 -- 172.21.15.148:0/400134679 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205800cb20 con 0x7f2060141ba0 2023-10-30T12:39:47.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20653ce640 5 monclient: authenticate success, global_id 22069 2023-10-30T12:39:47.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20653ce640 10 monclient: _renew_subs 2023-10-30T12:39:47.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20653ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:47.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20653ce640 1 -- 172.21.15.148:0/400134679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206006ec40 con 0x7f2060141ba0 2023-10-30T12:39:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 1 -- 172.21.15.148:0/400134679 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058003990 con 0x7f2060141ba0 2023-10-30T12:39:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20577fe640 10 monclient: dump: 2023-10-30T12:39:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:47.852 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:47.852 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.852 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:47.852 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:47.852 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:47.853 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:47.853 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:47.853 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:47.853 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:47.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f205dffb640 4 set_mon_vals no callback set 2023-10-30T12:39:47.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20653ce640 10 client.22069.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:47.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20653ce640 10 monclient: _renew_subs 2023-10-30T12:39:47.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20653ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:47.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20653ce640 1 -- 172.21.15.148:0/400134679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20600b2b50 con 0x7f2060141ba0 2023-10-30T12:39:47.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.844+0000 7f20653ce640 1 librados: init done 2023-10-30T12:39:47.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.845+0000 7f20653ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:47.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.845+0000 7f20653ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:47.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.845+0000 7f20653ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:47.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:47.845+0000 7f20653ce640 1 -- 172.21.15.148:0/400134679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2060070730 con 0x7f2060141ba0 2023-10-30T12:39:48.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.105+0000 7f20577fe640 1 -- 172.21.15.148:0/400134679 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f2058003d10 con 0x7f2060141ba0 2023-10-30T12:39:48.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.105+0000 7f20577fe640 1 --2- 172.21.15.148:0/400134679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f204006c810 0x7f204006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.106+0000 7f205effd640 1 --2- 172.21.15.148:0/400134679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f204006c810 0x7f204006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.106+0000 7f205effd640 10 monclient: get_auth_request con 0x7f204006c810 auth_method 0 2023-10-30T12:39:48.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.106+0000 7f205effd640 1 --2- 172.21.15.148:0/400134679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f204006c810 0x7f204006ecd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2050036000 tx=0x7f2050008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:48.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.106+0000 7f20577fe640 10 client.22069.objecter ms_handle_connect 0x7f204006c810 2023-10-30T12:39:48.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.108+0000 7f20577fe640 1 -- 172.21.15.148:0/400134679 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f2058059a70 con 0x7f2060141ba0 2023-10-30T12:39:48.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.108+0000 7f20577fe640 10 client.22069.objecter ms_dispatch 0x7f2060000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:48.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.108+0000 7f20577fe640 3 client.22069.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:48.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.108+0000 7f20577fe640 3 client.22069.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:48.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.108+0000 7f20577fe640 20 client.22069.objecter dump_active .. 0 homeless 2023-10-30T12:39:48.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.117+0000 7f20577fe640 1 -- 172.21.15.148:0/400134679 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2058059370 con 0x7f2060141ba0 2023-10-30T12:39:48.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.118+0000 7f20577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:48.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.118+0000 7f20577fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:48.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.314+0000 7f20653ce640 1 -- 172.21.15.148:0/400134679 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f20600af580 con 0x7f204006c810 2023-10-30T12:39:48.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.314+0000 7f20577fe640 1 -- 172.21.15.148:0/400134679 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2050006e10 con 0x7f204006c810 2023-10-30T12:39:48.316 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T12:39:48.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.316+0000 7f20653ce640 10 librados: watch_flush enter 2023-10-30T12:39:48.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.316+0000 7f20653ce640 10 librados: watch_flush exit 2023-10-30T12:39:48.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.316+0000 7f20653ce640 20 client.22069.objecter shutdown clearing up homeless session... 2023-10-30T12:39:48.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.316+0000 7f20653ce640 10 client.22069.objecter successfully canceled tick 2023-10-30T12:39:48.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 -- 172.21.15.148:0/400134679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f204006c810 msgr2=0x7f204006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 --2- 172.21.15.148:0/400134679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f204006c810 0x7f204006ecd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2050036000 tx=0x7f2050008030 comp rx=0 tx=0).stop 2023-10-30T12:39:48.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 10 monclient: shutdown 2023-10-30T12:39:48.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:48.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 -- 172.21.15.148:0/400134679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2060141ba0 msgr2=0x7f20600b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 --2- 172.21.15.148:0/400134679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2060141ba0 0x7f20600b2430 secure :-1 s=READY pgs=5436 cs=0 l=1 rev1=1 crypto rx=0x7f205800ec10 tx=0x7f205800c490 comp rx=0 tx=0).stop 2023-10-30T12:39:48.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 -- 172.21.15.148:0/400134679 shutdown_connections 2023-10-30T12:39:48.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 --2- 172.21.15.148:0/400134679 >> v2:172.21.15.148:6800/3205090179 conn(0x7f204006c810 0x7f204006ecd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 --2- 172.21.15.148:0/400134679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2060141ba0 0x7f20600b2430 unknown :-1 s=CLOSED pgs=5436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 --2- 172.21.15.148:0/400134679 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f206013ed20 0x7f20600701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 --2- 172.21.15.148:0/400134679 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f206006a0e0 0x7f206006faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 -- 172.21.15.148:0/400134679 >> 172.21.15.148:0/400134679 conn(0x7f206006b0b0 msgr2=0x7f20600b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:48.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 -- 172.21.15.148:0/400134679 shutdown_connections 2023-10-30T12:39:48.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 -- 172.21.15.148:0/400134679 wait complete. 2023-10-30T12:39:48.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.317+0000 7f20653ce640 1 librados: shutdown 2023-10-30T12:39:48.330 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T12:39:48.330 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:39:48.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.436+0000 7f59e5a9b640 10 monclient: get_monmap_and_config 2023-10-30T12:39:48.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.436+0000 7f59e5a9b640 10 monclient: build_initial_monmap 2023-10-30T12:39:48.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.436+0000 7f59e5a9b640 10 monclient: monmap: 2023-10-30T12:39:48.438 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:48.439 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:48.439 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:48.437375+0000 2023-10-30T12:39:48.439 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:48.437375+0000 2023-10-30T12:39:48.439 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:48.440 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:48.440 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:48.440 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:48.440 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:48.440 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:48.441 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:48.441 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 1 Processor -- start 2023-10-30T12:39:48.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 1 -- start start 2023-10-30T12:39:48.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 10 monclient: init 2023-10-30T12:39:48.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:48.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-10-30T12:39:48.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59e00aeb40 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:48.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59e00aabc0 0x7f59e013b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 10 monclient(hunting): picked mon.noname-b con 0x7f59e00aabc0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:48.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59e00ab040 0x7f59e013bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 10 monclient(hunting): picked mon.noname-c con 0x7f59e00ab040 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:48.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e013c2d0 con 0x7f59e00aeb40 2023-10-30T12:39:48.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:48.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e0067340 con 0x7f59e00ab040 2023-10-30T12:39:48.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:48.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e00aef20 con 0x7f59e00aabc0 2023-10-30T12:39:48.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:48.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.437+0000 7f59e5a9b640 10 monclient(hunting): authenticate will time out at 2880.642690s 2023-10-30T12:39:48.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59df7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f59e00aeb40 0x7f59e00ac8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42460/0 (socket says 172.21.15.148:42460) 2023-10-30T12:39:48.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59df7fe640 1 -- 172.21.15.148:0/3639240497 learned_addr learned my addr 172.21.15.148:0/3639240497 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:39:48.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59deffd640 1 --2- 172.21.15.148:0/3639240497 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59e00aabc0 0x7f59e013b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59deffd640 10 monclient(hunting): get_auth_request con 0x7f59e00aabc0 auth_method 0 2023-10-30T12:39:48.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:48.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59deffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59deffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59dffff640 1 --2- 172.21.15.148:0/3639240497 >> v2:172.21.15.150:3300/0 conn(0x7f59e00ab040 0x7f59e013bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59dffff640 10 monclient(hunting): get_auth_request con 0x7f59e00ab040 auth_method 0 2023-10-30T12:39:48.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:48.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59dffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.438+0000 7f59dffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.439+0000 7f59deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:48.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.439+0000 7f59deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:48.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.439+0000 7f59deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:48.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.439+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:48.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.439+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:48.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.439+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:48.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59de7fc640 1 -- 172.21.15.148:0/3639240497 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253026985 0 0) 0x7f59d4001110 con 0x7f59e00aeb40 2023-10-30T12:39:48.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59de7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:39:48.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59de7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59de7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59de7fc640 10 monclient(hunting): my global_id is 16254 2023-10-30T12:39:48.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59de7fc640 1 -- 172.21.15.148:0/3639240497 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c8003520 con 0x7f59e00aeb40 2023-10-30T12:39:48.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59deffd640 10 monclient(hunting): handle_auth_done global_id 17708 payload 274 2023-10-30T12:39:48.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59deffd640 1 -- 172.21.15.148:0/3639240497 >> v2:172.21.15.150:3300/0 conn(0x7f59e00ab040 msgr2=0x7f59e013bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:48.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59deffd640 1 --2- 172.21.15.148:0/3639240497 >> v2:172.21.15.150:3300/0 conn(0x7f59e00ab040 0x7f59e013bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59deffd640 1 -- 172.21.15.148:0/3639240497 >> v1:172.21.15.148:6789/0 conn(0x7f59e00aeb40 legacy=0x7f59e00ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59deffd640 10 monclient: _finish_hunting 0 2023-10-30T12:39:48.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59deffd640 1 monclient: found mon.noname-b 2023-10-30T12:39:48.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:48.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59deffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:48.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.440+0000 7f59deffd640 1 -- 172.21.15.148:0/3639240497 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e00ac130 con 0x7f59e00aabc0 2023-10-30T12:39:48.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.442+0000 7f59deffd640 1 --2- 172.21.15.148:0/3639240497 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59e00aabc0 0x7f59e013b850 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7f59d00097a0 tx=0x7f59d002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb28d0f1fcbb091f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:48.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 1 -- 172.21.15.148:0/3639240497 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0031700 con 0x7f59e00aabc0 2023-10-30T12:39:48.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:48.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:48.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 10 monclient: dump: 2023-10-30T12:39:48.458 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:48.458 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:48.459 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.459 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.459 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:48.459 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:48.459 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:48.460 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:48.460 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:48.460 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 10 monclient: _finish_auth 0 2023-10-30T12:39:48.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:39:48.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:48.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 1 -- 172.21.15.148:0/3639240497 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d0031870 con 0x7f59e00aabc0 2023-10-30T12:39:48.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:48.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 1 -- 172.21.15.148:0/3639240497 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0031c10 con 0x7f59e00aabc0 2023-10-30T12:39:48.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59e5a9b640 5 monclient: authenticate success, global_id 17708 2023-10-30T12:39:48.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59e5a9b640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:48.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59e5a9b640 4 set_mon_vals no callback set 2023-10-30T12:39:48.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:48.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:48.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.446+0000 7f59de7fc640 10 monclient: dump: 2023-10-30T12:39:48.462 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:48.463 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:48.463 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.463 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.463 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:48.463 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:48.463 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:48.464 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:48.464 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:48.464 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.447+0000 7f59e5a9b640 10 monclient: shutdown 2023-10-30T12:39:48.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.447+0000 7f59e5a9b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:48.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.447+0000 7f59e5a9b640 1 -- 172.21.15.148:0/3639240497 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59e00aabc0 msgr2=0x7f59e013b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.447+0000 7f59e5a9b640 1 --2- 172.21.15.148:0/3639240497 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59e00aabc0 0x7f59e013b850 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7f59d00097a0 tx=0x7f59d002b7a0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.447+0000 7f59e5a9b640 1 -- 172.21.15.148:0/3639240497 shutdown_connections 2023-10-30T12:39:48.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.447+0000 7f59e5a9b640 1 --2- 172.21.15.148:0/3639240497 >> v2:172.21.15.150:3300/0 conn(0x7f59e00ab040 0x7f59e013bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.447+0000 7f59e5a9b640 1 --2- 172.21.15.148:0/3639240497 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59e00aabc0 0x7f59e013b850 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.447+0000 7f59e5a9b640 1 -- 172.21.15.148:0/3639240497 >> 172.21.15.148:0/3639240497 conn(0x7f59e00b3260 msgr2=0x7f59e00b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:48.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.447+0000 7f59e5a9b640 1 -- 172.21.15.148:0/3639240497 shutdown_connections 2023-10-30T12:39:48.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.447+0000 7f59e5a9b640 1 -- 172.21.15.148:0/3639240497 wait complete. 2023-10-30T12:39:48.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.448+0000 7f59e5a9b640 10 monclient: build_initial_monmap 2023-10-30T12:39:48.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.448+0000 7f59e5a9b640 10 monclient: monmap: 2023-10-30T12:39:48.467 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:48.467 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:48.467 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:48.467 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:48.467 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:48.468 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:48.468 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:48.468 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:48.468 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:48.468 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.448+0000 7f59e5a9b640 1 librados: starting msgr at 2023-10-30T12:39:48.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.448+0000 7f59e5a9b640 1 librados: starting objecter 2023-10-30T12:39:48.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.448+0000 7f59e5a9b640 1 Processor -- start 2023-10-30T12:39:48.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.448+0000 7f59e5a9b640 1 -- start start 2023-10-30T12:39:48.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.448+0000 7f59e5a9b640 1 librados: setting wanted keys 2023-10-30T12:39:48.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.448+0000 7f59e5a9b640 1 librados: calling monclient init 2023-10-30T12:39:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.448+0000 7f59e5a9b640 10 monclient: init 2023-10-30T12:39:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:39:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59e00aabc0 0x7f59e013d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 10 monclient(hunting): picked mon.noname-b con 0x7f59e00aabc0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:48.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59e00ab040 0x7f59e013f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 10 monclient(hunting): picked mon.noname-a con 0x7f59e00ab040 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:48.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59e00aeb40 0x7f59e013c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 10 monclient(hunting): picked mon.noname-c con 0x7f59e00aeb40 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:48.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:48.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e00acb90 con 0x7f59e00ab040 2023-10-30T12:39:48.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:48.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e013f890 con 0x7f59e00aabc0 2023-10-30T12:39:48.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:48.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e0142a60 con 0x7f59e00aeb40 2023-10-30T12:39:48.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:48.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59e5a9b640 10 monclient(hunting): authenticate will time out at 2880.654358s 2023-10-30T12:39:48.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.449+0000 7f59dffff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59e00aeb40 0x7f59e013c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59deffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59e00ab040 0x7f59e013f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59e00aabc0 0x7f59e013d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59e00aabc0 0x7f59e013d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:55562/0 (socket says 172.21.15.148:55562) 2023-10-30T12:39:48.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59df7fe640 1 -- 172.21.15.148:0/2033276217 learned_addr learned my addr 172.21.15.148:0/2033276217 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:48.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59dffff640 10 monclient(hunting): get_auth_request con 0x7f59e00aeb40 auth_method 0 2023-10-30T12:39:48.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:48.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59dffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59dffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59df7fe640 10 monclient(hunting): get_auth_request con 0x7f59e00aabc0 auth_method 0 2023-10-30T12:39:48.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:48.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59df7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59df7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59deffd640 10 monclient(hunting): get_auth_request con 0x7f59e00ab040 auth_method 0 2023-10-30T12:39:48.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:48.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59deffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59deffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:48.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:48.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:48.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:48.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.450+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:48.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.451+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:48.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.451+0000 7f59deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:48.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.451+0000 7f59deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:48.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.451+0000 7f59deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:48.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.453+0000 7f59df7fe640 10 monclient(hunting): handle_auth_done global_id 22075 payload 824 2023-10-30T12:39:48.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.453+0000 7f59df7fe640 1 -- 172.21.15.148:0/2033276217 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59e00aeb40 msgr2=0x7f59e013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.453+0000 7f59df7fe640 1 --2- 172.21.15.148:0/2033276217 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59e00aeb40 0x7f59e013c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.453+0000 7f59df7fe640 1 -- 172.21.15.148:0/2033276217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59e00ab040 msgr2=0x7f59e013f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.453+0000 7f59df7fe640 1 --2- 172.21.15.148:0/2033276217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59e00ab040 0x7f59e013f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.453+0000 7f59df7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:39:48.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.453+0000 7f59df7fe640 1 monclient: found mon.noname-b 2023-10-30T12:39:48.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.453+0000 7f59df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:48.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.453+0000 7f59df7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:39:48.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.453+0000 7f59df7fe640 1 -- 172.21.15.148:0/2033276217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e013cf80 con 0x7f59e00aabc0 2023-10-30T12:39:48.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.455+0000 7f59df7fe640 1 --2- 172.21.15.148:0/2033276217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59e00aabc0 0x7f59e013d4a0 secure :-1 s=READY pgs=5437 cs=0 l=1 rev1=1 crypto rx=0x7f59d400c730 tx=0x7f59d400cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:48.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.455+0000 7f59c7fff640 10 client.?.objecter ms_handle_connect 0x7f59e00aabc0 2023-10-30T12:39:48.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.455+0000 7f59c7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:48.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.458+0000 7f59c7fff640 1 -- 172.21.15.148:0/2033276217 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4005ab0 con 0x7f59e00aabc0 2023-10-30T12:39:48.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:48.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:48.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 10 monclient: dump: 2023-10-30T12:39:48.484 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:48.484 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:48.484 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.485 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.485 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:48.485 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:48.485 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:48.485 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:48.486 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:48.486 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 10 monclient: _finish_auth 0 2023-10-30T12:39:48.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:39:48.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:48.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 1 -- 172.21.15.148:0/2033276217 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d400e040 con 0x7f59e00aabc0 2023-10-30T12:39:48.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59e5a9b640 5 monclient: authenticate success, global_id 22075 2023-10-30T12:39:48.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59e5a9b640 10 monclient: _renew_subs 2023-10-30T12:39:48.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59e5a9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:48.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e013c9d0 con 0x7f59e00aabc0 2023-10-30T12:39:48.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:48.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 1 -- 172.21.15.148:0/2033276217 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4005e40 con 0x7f59e00aabc0 2023-10-30T12:39:48.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:48.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:48.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59c7fff640 10 monclient: dump: 2023-10-30T12:39:48.488 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:48.489 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:48.489 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.489 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.489 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:48.489 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:48.490 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:48.490 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:48.490 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:48.490 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59ddffb640 4 set_mon_vals no callback set 2023-10-30T12:39:48.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59e5a9b640 10 client.22075.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:48.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59e5a9b640 10 monclient: _renew_subs 2023-10-30T12:39:48.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59e5a9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:48.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e01d26d0 con 0x7f59e00aabc0 2023-10-30T12:39:48.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59e5a9b640 1 librados: init done 2023-10-30T12:39:48.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.459+0000 7f59e5a9b640 10 client.22075.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:48.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.460+0000 7f59e5a9b640 15 client.22075.objecter _session_command_op_assign -1 1 2023-10-30T12:39:48.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.460+0000 7f59e5a9b640 20 client.22075.objecter _get_session osd=-1 returning homeless 2023-10-30T12:39:48.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.460+0000 7f59e5a9b640 10 client.22075.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:48.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.460+0000 7f59e5a9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:48.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.460+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f59e00aa700 con 0x7f59e00aabc0 2023-10-30T12:39:48.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.479+0000 7f59c7fff640 1 -- 172.21.15.148:0/2033276217 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f59d4007070 con 0x7f59e00aabc0 2023-10-30T12:39:48.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.479+0000 7f59c7fff640 1 --2- 172.21.15.148:0/2033276217 >> v2:172.21.15.148:6800/3205090179 conn(0x7f59cc06c7c0 0x7f59cc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.479+0000 7f59deffd640 1 --2- 172.21.15.148:0/2033276217 >> v2:172.21.15.148:6800/3205090179 conn(0x7f59cc06c7c0 0x7f59cc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.479+0000 7f59deffd640 10 monclient: get_auth_request con 0x7f59cc06c7c0 auth_method 0 2023-10-30T12:39:48.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.479+0000 7f59deffd640 1 --2- 172.21.15.148:0/2033276217 >> v2:172.21.15.148:6800/3205090179 conn(0x7f59cc06c7c0 0x7f59cc06ec80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f59d0035000 tx=0x7f59d0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:48.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.479+0000 7f59c7fff640 10 client.22075.objecter ms_handle_connect 0x7f59cc06c7c0 2023-10-30T12:39:48.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 1 -- 172.21.15.148:0/2033276217 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f59d4007d60 con 0x7f59e00aabc0 2023-10-30T12:39:48.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 10 client.22075.objecter ms_dispatch 0x7f59e0000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:48.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 3 client.22075.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:48.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 3 client.22075.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:48.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 10 client.22075.objecter checking command 1 2023-10-30T12:39:48.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 1 --2- 172.21.15.148:0/2033276217 >> v2:172.21.15.148:6813/912412250 conn(0x7f59cc073ae0 0x7f59cc075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 20 client.22075.objecter _get_session s=0x7f59d4005e40 osd=1 3 2023-10-30T12:39:48.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 20 client.22075.objecter put_session s=0x7f59d4005e40 osd=1 3 2023-10-30T12:39:48.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 15 client.22075.objecter _session_command_op_remove -1 1 2023-10-30T12:39:48.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 20 client.22075.objecter _get_session s=0x7f59d4005e40 osd=1 3 2023-10-30T12:39:48.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 20 client.22075.objecter get_session s=0x7f59d4005e40 osd=1 3 2023-10-30T12:39:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 15 client.22075.objecter _session_command_op_assign 1 1 2023-10-30T12:39:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 20 client.22075.objecter put_session s=0x7f59d4005e40 osd=1 4 2023-10-30T12:39:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 10 client.22075.objecter _send_command 1 2023-10-30T12:39:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 1 -- 172.21.15.148:0/2033276217 --> v2:172.21.15.148:6813/912412250 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59d4005ab0 con 0x7f59cc073ae0 2023-10-30T12:39:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 20 client.22075.objecter dump_active .. 0 homeless 2023-10-30T12:39:48.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 1 -- 172.21.15.148:0/2033276217 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59e013f890 con 0x7f59e00aabc0 2023-10-30T12:39:48.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59c7fff640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:39:48.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59dffff640 1 --2- 172.21.15.148:0/2033276217 >> v2:172.21.15.148:6813/912412250 conn(0x7f59cc073ae0 0x7f59cc075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.481+0000 7f59dffff640 10 monclient: get_auth_request con 0x7f59cc073ae0 auth_method 0 2023-10-30T12:39:48.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.484+0000 7f59dffff640 1 --2- 172.21.15.148:0/2033276217 >> v2:172.21.15.148:6813/912412250 conn(0x7f59cc073ae0 0x7f59cc075f80 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:48.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.484+0000 7f59c7fff640 10 client.22075.objecter ms_handle_connect 0x7f59cc073ae0 2023-10-30T12:39:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.506+0000 7f59c7fff640 1 -- 172.21.15.148:0/2033276217 <== osd.1 v2:172.21.15.148:6813/912412250 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f59d4005ab0 con 0x7f59cc073ae0 2023-10-30T12:39:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.506+0000 7f59c7fff640 10 client.22075.objecter ms_dispatch 0x7f59e0000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:39:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.506+0000 7f59c7fff640 10 client.22075.objecter _finish_command 1 = system:0 2023-10-30T12:39:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.506+0000 7f59c7fff640 20 client.22075.objecter put_session s=0x7f59d4005e40 osd=1 4 2023-10-30T12:39:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.506+0000 7f59c7fff640 15 client.22075.objecter _session_command_op_remove 1 1 2023-10-30T12:39:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 10 client.22075.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:39:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 15 client.22075.objecter _session_command_op_assign -1 2 2023-10-30T12:39:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 20 client.22075.objecter _get_session s=0x7f59d4005e40 osd=1 3 2023-10-30T12:39:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 20 client.22075.objecter put_session s=0x7f59d4005e40 osd=1 3 2023-10-30T12:39:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 20 client.22075.objecter _get_session s=0x7f59d4005e40 osd=1 3 2023-10-30T12:39:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 15 client.22075.objecter _session_command_op_remove -1 2 2023-10-30T12:39:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 20 client.22075.objecter get_session s=0x7f59d4005e40 osd=1 3 2023-10-30T12:39:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 15 client.22075.objecter _session_command_op_assign 1 2 2023-10-30T12:39:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 20 client.22075.objecter put_session s=0x7f59d4005e40 osd=1 4 2023-10-30T12:39:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 10 client.22075.objecter _send_command 2 2023-10-30T12:39:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.518+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 --> v2:172.21.15.148:6813/912412250 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f59e013c9d0 con 0x7f59cc073ae0 2023-10-30T12:39:48.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59c7fff640 1 -- 172.21.15.148:0/2033276217 <== osd.1 v2:172.21.15.148:6813/912412250 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f59e013c9d0 con 0x7f59cc073ae0 2023-10-30T12:39:48.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59c7fff640 10 client.22075.objecter ms_dispatch 0x7f59e0000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:39:48.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59c7fff640 10 client.22075.objecter _finish_command 2 = system:0 2023-10-30T12:39:48.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59c7fff640 20 client.22075.objecter put_session s=0x7f59d4005e40 osd=1 4 2023-10-30T12:39:48.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59c7fff640 15 client.22075.objecter _session_command_op_remove 1 2 2023-10-30T12:39:48.535 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T12:39:48.535 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:39:48.535 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T12:39:48.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 10 librados: watch_flush enter 2023-10-30T12:39:48.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 10 librados: watch_flush exit 2023-10-30T12:39:48.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 10 client.22075.objecter close_session for osd.1 2023-10-30T12:39:48.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 >> v2:172.21.15.148:6813/912412250 conn(0x7f59cc073ae0 msgr2=0x7f59cc075f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 1 --2- 172.21.15.148:0/2033276217 >> v2:172.21.15.148:6813/912412250 conn(0x7f59cc073ae0 0x7f59cc075f80 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 20 client.22075.objecter put_session s=0x7f59d4005e40 osd=1 1 2023-10-30T12:39:48.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 20 client.22075.objecter shutdown clearing up homeless session... 2023-10-30T12:39:48.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 10 client.22075.objecter successfully canceled tick 2023-10-30T12:39:48.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 >> v2:172.21.15.148:6800/3205090179 conn(0x7f59cc06c7c0 msgr2=0x7f59cc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 1 --2- 172.21.15.148:0/2033276217 >> v2:172.21.15.148:6800/3205090179 conn(0x7f59cc06c7c0 0x7f59cc06ec80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f59d0035000 tx=0x7f59d0008030 comp rx=0 tx=0).stop 2023-10-30T12:39:48.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 10 monclient: shutdown 2023-10-30T12:39:48.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:48.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59e00aabc0 msgr2=0x7f59e013d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 1 --2- 172.21.15.148:0/2033276217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59e00aabc0 0x7f59e013d4a0 secure :-1 s=READY pgs=5437 cs=0 l=1 rev1=1 crypto rx=0x7f59d400c730 tx=0x7f59d400cbf0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59dffff640 1 -- 172.21.15.148:0/2033276217 reap_dead start 2023-10-30T12:39:48.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 shutdown_connections 2023-10-30T12:39:48.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.532+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 >> 172.21.15.148:0/2033276217 conn(0x7f59e00b3260 msgr2=0x7f59e00a86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:48.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.533+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 shutdown_connections 2023-10-30T12:39:48.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.533+0000 7f59e5a9b640 1 -- 172.21.15.148:0/2033276217 wait complete. 2023-10-30T12:39:48.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.533+0000 7f59e5a9b640 1 librados: shutdown 2023-10-30T12:39:48.545 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T12:39:48.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.643+0000 7fc3c39c5640 10 monclient: get_monmap_and_config 2023-10-30T12:39:48.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.643+0000 7fc3c39c5640 10 monclient: build_initial_monmap 2023-10-30T12:39:48.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.644+0000 7fc3c39c5640 10 monclient: monmap: 2023-10-30T12:39:48.647 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:48.647 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:48.647 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:48.645045+0000 2023-10-30T12:39:48.647 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:48.645045+0000 2023-10-30T12:39:48.648 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:48.648 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:48.648 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:48.648 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:48.649 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:48.649 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:48.649 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:48.649 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.644+0000 7fc3c39c5640 1 Processor -- start 2023-10-30T12:39:48.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.644+0000 7fc3c39c5640 1 -- start start 2023-10-30T12:39:48.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.644+0000 7fc3c39c5640 10 monclient: init 2023-10-30T12:39:48.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:48.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-10-30T12:39:48.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc3bc06bb80 0x7fc3bc06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 10 monclient(hunting): picked mon.noname-a con 0x7fc3bc06bb80 addr v2:172.21.15.148:3300/0 2023-10-30T12:39:48.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3bc0721b0 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:48.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc3bc0afd50 0x7fc3bc1458e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 10 monclient(hunting): picked mon.noname-c con 0x7fc3bc0afd50 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:48.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc071a90 con 0x7fc3bc0721b0 2023-10-30T12:39:48.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:48.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3bc0670a0 con 0x7fc3bc06bb80 2023-10-30T12:39:48.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:48.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3bc06bf60 con 0x7fc3bc0afd50 2023-10-30T12:39:48.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:48.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c39c5640 10 monclient(hunting): authenticate will time out at 2880.850272s 2023-10-30T12:39:48.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.645+0000 7fc3c1761640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc3bc06bb80 0x7fc3bc06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1f62640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc3bc0afd50 0x7fc3bc1458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1761640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc3bc06bb80 0x7fc3bc06cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55206/0 (socket says 172.21.15.148:55206) 2023-10-30T12:39:48.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1761640 1 -- 172.21.15.148:0/940717962 learned_addr learned my addr 172.21.15.148:0/940717962 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:48.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1761640 10 monclient(hunting): get_auth_request con 0x7fc3bc06bb80 auth_method 0 2023-10-30T12:39:48.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:48.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1761640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1761640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1f62640 10 monclient(hunting): get_auth_request con 0x7fc3bc0afd50 auth_method 0 2023-10-30T12:39:48.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:48.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1f62640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1f62640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1761640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:48.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:48.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.646+0000 7fc3c1761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:48.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.647+0000 7fc3c1f62640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:48.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.647+0000 7fc3c1f62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:48.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.647+0000 7fc3c1f62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:48.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.648+0000 7fc3c1761640 10 monclient(hunting): handle_auth_done global_id 16260 payload 274 2023-10-30T12:39:48.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.648+0000 7fc3c1761640 1 -- 172.21.15.148:0/940717962 >> v2:172.21.15.150:3300/0 conn(0x7fc3bc0afd50 msgr2=0x7fc3bc1458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:48.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.648+0000 7fc3c1761640 1 --2- 172.21.15.148:0/940717962 >> v2:172.21.15.150:3300/0 conn(0x7fc3bc0afd50 0x7fc3bc1458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.648+0000 7fc3c1761640 1 -- 172.21.15.148:0/940717962 >> v1:172.21.15.150:6789/0 conn(0x7fc3bc0721b0 legacy=0x7fc3bc1451d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:48.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.648+0000 7fc3c1761640 10 monclient: _finish_hunting 0 2023-10-30T12:39:48.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.648+0000 7fc3c1761640 1 monclient: found mon.noname-a 2023-10-30T12:39:48.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.648+0000 7fc3c1761640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:48.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.648+0000 7fc3c1761640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:39:48.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.648+0000 7fc3c1761640 1 -- 172.21.15.148:0/940717962 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc06fd60 con 0x7fc3bc06bb80 2023-10-30T12:39:48.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.650+0000 7fc3c1761640 1 --2- 172.21.15.148:0/940717962 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3bc06bb80 0x7fc3bc06cbe0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0ab730 tx=0x7fc3ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c27921d0368072 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:48.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 1 -- 172.21.15.148:0/940717962 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac0317e0 con 0x7fc3bc06bb80 2023-10-30T12:39:48.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:48.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:39:48.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 10 monclient: dump: 2023-10-30T12:39:48.665 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:48.665 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:48.665 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.665 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.666 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:48.666 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:48.666 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:48.666 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:48.666 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:48.667 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 10 monclient: _finish_auth 0 2023-10-30T12:39:48.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:39:48.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:48.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 1 -- 172.21.15.148:0/940717962 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ac031950 con 0x7fc3bc06bb80 2023-10-30T12:39:48.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3c39c5640 5 monclient: authenticate success, global_id 16260 2023-10-30T12:39:48.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3c39c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:39:48.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:48.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3c39c5640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:48.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3c39c5640 4 set_mon_vals no callback set 2023-10-30T12:39:48.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 1 -- 172.21.15.148:0/940717962 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac031cf0 con 0x7fc3bc06bb80 2023-10-30T12:39:48.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:48.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:39:48.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.653+0000 7fc3b3fff640 10 monclient: dump: 2023-10-30T12:39:48.669 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:48.670 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:48.670 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.670 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.670 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:48.670 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:48.671 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:48.671 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:48.671 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:48.671 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.654+0000 7fc3c39c5640 10 monclient: shutdown 2023-10-30T12:39:48.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.654+0000 7fc3c39c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:48.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.654+0000 7fc3c39c5640 1 -- 172.21.15.148:0/940717962 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3bc06bb80 msgr2=0x7fc3bc06cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.654+0000 7fc3c39c5640 1 --2- 172.21.15.148:0/940717962 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3bc06bb80 0x7fc3bc06cbe0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0ab730 tx=0x7fc3ac02b880 comp rx=0 tx=0).stop 2023-10-30T12:39:48.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.654+0000 7fc3c39c5640 1 -- 172.21.15.148:0/940717962 shutdown_connections 2023-10-30T12:39:48.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.654+0000 7fc3c39c5640 1 --2- 172.21.15.148:0/940717962 >> v2:172.21.15.150:3300/0 conn(0x7fc3bc0afd50 0x7fc3bc1458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.654+0000 7fc3c39c5640 1 --2- 172.21.15.148:0/940717962 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3bc06bb80 0x7fc3bc06cbe0 unknown :-1 s=CLOSED pgs=3228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.654+0000 7fc3c39c5640 1 -- 172.21.15.148:0/940717962 >> 172.21.15.148:0/940717962 conn(0x7fc3bc0b54e0 msgr2=0x7fc3bc0b58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:48.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.654+0000 7fc3c39c5640 1 -- 172.21.15.148:0/940717962 shutdown_connections 2023-10-30T12:39:48.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.654+0000 7fc3c39c5640 1 -- 172.21.15.148:0/940717962 wait complete. 2023-10-30T12:39:48.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.655+0000 7fc3c39c5640 10 monclient: build_initial_monmap 2023-10-30T12:39:48.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.655+0000 7fc3c39c5640 10 monclient: monmap: 2023-10-30T12:39:48.673 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:48.674 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:48.674 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:48.674 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:48.674 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:48.674 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:48.675 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:48.675 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:48.675 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:48.675 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.655+0000 7fc3c39c5640 1 librados: starting msgr at 2023-10-30T12:39:48.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.655+0000 7fc3c39c5640 1 librados: starting objecter 2023-10-30T12:39:48.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.655+0000 7fc3c39c5640 1 Processor -- start 2023-10-30T12:39:48.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.655+0000 7fc3c39c5640 1 -- start start 2023-10-30T12:39:48.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.655+0000 7fc3c39c5640 1 librados: setting wanted keys 2023-10-30T12:39:48.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.655+0000 7fc3c39c5640 1 librados: calling monclient init 2023-10-30T12:39:48.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.655+0000 7fc3c39c5640 10 monclient: init 2023-10-30T12:39:48.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:48.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:39:48.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3bc06bb80 0x7fc3bc0a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 10 monclient(hunting): picked mon.noname-a con 0x7fc3bc06bb80 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:48.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3bc0721b0 0x7fc3bc13a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 10 monclient(hunting): picked mon.noname-b con 0x7fc3bc0721b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:48.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc3bc0afd50 0x7fc3bc13acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 10 monclient(hunting): picked mon.noname-c con 0x7fc3bc0afd50 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:48.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:48.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc0abd40 con 0x7fc3bc06bb80 2023-10-30T12:39:48.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:48.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc142f20 con 0x7fc3bc0721b0 2023-10-30T12:39:48.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:48.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc13b210 con 0x7fc3bc0afd50 2023-10-30T12:39:48.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:48.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c39c5640 10 monclient(hunting): authenticate will time out at 2880.861000s 2023-10-30T12:39:48.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c1761640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3bc06bb80 0x7fc3bc0a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c1f62640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc3bc0afd50 0x7fc3bc13acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c1761640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3bc06bb80 0x7fc3bc0a94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55216/0 (socket says 172.21.15.148:55216) 2023-10-30T12:39:48.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c1761640 1 -- 172.21.15.148:0/2718668661 learned_addr learned my addr 172.21.15.148:0/2718668661 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:48.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c1761640 10 monclient(hunting): get_auth_request con 0x7fc3bc06bb80 auth_method 0 2023-10-30T12:39:48.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c1761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:48.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c1761640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.656+0000 7fc3c1761640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c0f60640 1 --2- 172.21.15.148:0/2718668661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3bc0721b0 0x7fc3bc13a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c1f62640 10 monclient(hunting): get_auth_request con 0x7fc3bc0afd50 auth_method 0 2023-10-30T12:39:48.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c1f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:48.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c1f62640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c1f62640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c0f60640 10 monclient(hunting): get_auth_request con 0x7fc3bc0721b0 auth_method 0 2023-10-30T12:39:48.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c0f60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:48.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c0f60640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:48.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c0f60640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:48.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c1761640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:48.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c1761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:48.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c1761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:48.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c1f62640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:48.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c1f62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:48.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c1f62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:48.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c0f60640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:48.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c0f60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:48.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.657+0000 7fc3c0f60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:48.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.660+0000 7fc3c0f60640 10 monclient(hunting): handle_auth_done global_id 22081 payload 824 2023-10-30T12:39:48.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.660+0000 7fc3c0f60640 1 -- 172.21.15.148:0/2718668661 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc3bc0afd50 msgr2=0x7fc3bc13acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.660+0000 7fc3c0f60640 1 --2- 172.21.15.148:0/2718668661 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc3bc0afd50 0x7fc3bc13acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.660+0000 7fc3c0f60640 1 -- 172.21.15.148:0/2718668661 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3bc06bb80 msgr2=0x7fc3bc0a94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:48.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.660+0000 7fc3c0f60640 1 --2- 172.21.15.148:0/2718668661 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3bc06bb80 0x7fc3bc0a94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:48.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.660+0000 7fc3c0f60640 10 monclient: _finish_hunting 0 2023-10-30T12:39:48.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.660+0000 7fc3c0f60640 1 monclient: found mon.noname-b 2023-10-30T12:39:48.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.660+0000 7fc3c0f60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:48.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.660+0000 7fc3c0f60640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:39:48.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.660+0000 7fc3c0f60640 1 -- 172.21.15.148:0/2718668661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc0ab320 con 0x7fc3bc0721b0 2023-10-30T12:39:48.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.662+0000 7fc3c0f60640 1 --2- 172.21.15.148:0/2718668661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3bc0721b0 0x7fc3bc13a790 secure :-1 s=READY pgs=5439 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800c410 tx=0x7fc3b800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:48.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.662+0000 7fc3b1ffb640 10 client.?.objecter ms_handle_connect 0x7fc3bc0721b0 2023-10-30T12:39:48.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.662+0000 7fc3b1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:48.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 1 -- 172.21.15.148:0/2718668661 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8005230 con 0x7fc3bc0721b0 2023-10-30T12:39:48.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:48.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:48.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 10 monclient: dump: 2023-10-30T12:39:48.689 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:48.689 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:48.690 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.690 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.690 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:48.690 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:48.690 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:48.691 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:48.691 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:48.691 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:39:48.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:39:48.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:48.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 1 -- 172.21.15.148:0/2718668661 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b800e040 con 0x7fc3bc0721b0 2023-10-30T12:39:48.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:48.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 1 -- 172.21.15.148:0/2718668661 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b80045e0 con 0x7fc3bc0721b0 2023-10-30T12:39:48.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.665+0000 7fc3b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:48.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:48.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3b1ffb640 10 monclient: dump: 2023-10-30T12:39:48.693 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:48.693 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:48.693 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.693 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:48.694 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:48.694 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:48.694 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:48.694 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:48.694 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:48.695 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:48.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3c39c5640 5 monclient: authenticate success, global_id 22081 2023-10-30T12:39:48.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3c39c5640 10 monclient: _renew_subs 2023-10-30T12:39:48.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3c39c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:48.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3c39c5640 1 -- 172.21.15.148:0/2718668661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3bc1448d0 con 0x7fc3bc0721b0 2023-10-30T12:39:48.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3b37fe640 4 set_mon_vals no callback set 2023-10-30T12:39:48.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3c39c5640 10 client.22081.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:48.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3c39c5640 10 monclient: _renew_subs 2023-10-30T12:39:48.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3c39c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:48.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3c39c5640 1 -- 172.21.15.148:0/2718668661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3bc0a8750 con 0x7fc3bc0721b0 2023-10-30T12:39:48.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.666+0000 7fc3c39c5640 1 librados: init done 2023-10-30T12:39:48.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.667+0000 7fc3c39c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:48.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.667+0000 7fc3c39c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:48.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.667+0000 7fc3c39c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:48.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.667+0000 7fc3c39c5640 1 -- 172.21.15.148:0/2718668661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3bc0a9c10 con 0x7fc3bc0721b0 2023-10-30T12:39:48.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.684+0000 7fc3b1ffb640 1 -- 172.21.15.148:0/2718668661 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fc3b8007020 con 0x7fc3bc0721b0 2023-10-30T12:39:48.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.685+0000 7fc3b1ffb640 1 --2- 172.21.15.148:0/2718668661 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc3a006c810 0x7fc3a006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:48.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.685+0000 7fc3c1761640 1 --2- 172.21.15.148:0/2718668661 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc3a006c810 0x7fc3a006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:48.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.685+0000 7fc3c1761640 10 monclient: get_auth_request con 0x7fc3a006c810 auth_method 0 2023-10-30T12:39:48.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.685+0000 7fc3c1761640 1 --2- 172.21.15.148:0/2718668661 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc3a006c810 0x7fc3a006ecd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac02b8c0 tx=0x7fc3ac03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:48.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.685+0000 7fc3b1ffb640 10 client.22081.objecter ms_handle_connect 0x7fc3a006c810 2023-10-30T12:39:48.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.687+0000 7fc3b1ffb640 1 -- 172.21.15.148:0/2718668661 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7fc3b80573c0 con 0x7fc3bc0721b0 2023-10-30T12:39:48.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.687+0000 7fc3b1ffb640 10 client.22081.objecter ms_dispatch 0x7fc3bc000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:48.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.687+0000 7fc3b1ffb640 3 client.22081.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:48.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.687+0000 7fc3b1ffb640 3 client.22081.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:48.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.687+0000 7fc3b1ffb640 20 client.22081.objecter dump_active .. 0 homeless 2023-10-30T12:39:48.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.921+0000 7fc3b1ffb640 1 -- 172.21.15.148:0/2718668661 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc3b805b460 con 0x7fc3bc0721b0 2023-10-30T12:39:48.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.921+0000 7fc3b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:48.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:48.921+0000 7fc3b1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:49.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.103+0000 7fc3c39c5640 1 -- 172.21.15.148:0/2718668661 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3bc070320 con 0x7fc3a006c810 2023-10-30T12:39:49.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.103+0000 7fc3b1ffb640 1 -- 172.21.15.148:0/2718668661 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc3ac0317b0 con 0x7fc3a006c810 2023-10-30T12:39:49.105 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T12:39:49.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 10 librados: watch_flush enter 2023-10-30T12:39:49.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 10 librados: watch_flush exit 2023-10-30T12:39:49.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 20 client.22081.objecter shutdown clearing up homeless session... 2023-10-30T12:39:49.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 10 client.22081.objecter successfully canceled tick 2023-10-30T12:39:49.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 -- 172.21.15.148:0/2718668661 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc3a006c810 msgr2=0x7fc3a006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 --2- 172.21.15.148:0/2718668661 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc3a006c810 0x7fc3a006ecd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac02b8c0 tx=0x7fc3ac03d040 comp rx=0 tx=0).stop 2023-10-30T12:39:49.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 10 monclient: shutdown 2023-10-30T12:39:49.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:49.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 -- 172.21.15.148:0/2718668661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3bc0721b0 msgr2=0x7fc3bc13a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 --2- 172.21.15.148:0/2718668661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3bc0721b0 0x7fc3bc13a790 secure :-1 s=READY pgs=5439 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800c410 tx=0x7fc3b800c8d0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 -- 172.21.15.148:0/2718668661 shutdown_connections 2023-10-30T12:39:49.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 --2- 172.21.15.148:0/2718668661 >> v2:172.21.15.148:6800/3205090179 conn(0x7fc3a006c810 0x7fc3a006ecd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 --2- 172.21.15.148:0/2718668661 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fc3bc0afd50 0x7fc3bc13acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 --2- 172.21.15.148:0/2718668661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3bc0721b0 0x7fc3bc13a790 unknown :-1 s=CLOSED pgs=5439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 --2- 172.21.15.148:0/2718668661 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3bc06bb80 0x7fc3bc0a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 -- 172.21.15.148:0/2718668661 >> 172.21.15.148:0/2718668661 conn(0x7fc3bc0b54e0 msgr2=0x7fc3bc145a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:49.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 -- 172.21.15.148:0/2718668661 shutdown_connections 2023-10-30T12:39:49.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 -- 172.21.15.148:0/2718668661 wait complete. 2023-10-30T12:39:49.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.105+0000 7fc3c39c5640 1 librados: shutdown 2023-10-30T12:39:49.120 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T12:39:49.120 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:39:49.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.221+0000 7f20b1d1d640 10 monclient: get_monmap_and_config 2023-10-30T12:39:49.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.221+0000 7f20b1d1d640 10 monclient: build_initial_monmap 2023-10-30T12:39:49.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.221+0000 7f20b1d1d640 10 monclient: monmap: 2023-10-30T12:39:49.224 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:49.224 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:49.224 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:49.222718+0000 2023-10-30T12:39:49.224 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:49.222718+0000 2023-10-30T12:39:49.224 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:49.225 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:49.225 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:49.225 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:49.225 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:49.225 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:49.225 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:49.226 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.222+0000 7f20b1d1d640 1 Processor -- start 2023-10-30T12:39:49.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.222+0000 7f20b1d1d640 1 -- start start 2023-10-30T12:39:49.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.222+0000 7f20b1d1d640 10 monclient: init 2023-10-30T12:39:49.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.222+0000 7f20b1d1d640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:49.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.222+0000 7f20b1d1d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T12:39:49.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.222+0000 7f20b1d1d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20ac065d20 0x7f20ac071580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.222+0000 7f20b1d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f20ac065d20 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:49.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.222+0000 7f20b1d1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20ac06d9d0 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:49.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20b1d1d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 0x7f20ac071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20b1d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f20ac0afda0 addr v2:172.21.15.148:3300/0 2023-10-30T12:39:49.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20b1d1d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac0af3d0 con 0x7f20ac06d9d0 2023-10-30T12:39:49.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20b1d1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20b1d1d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ac067110 con 0x7f20ac0afda0 2023-10-30T12:39:49.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20b1d1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20b1d1d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ac066100 con 0x7f20ac065d20 2023-10-30T12:39:49.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20b1d1d640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:49.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20b1d1d640 10 monclient(hunting): authenticate will time out at 2881.427920s 2023-10-30T12:39:49.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20abfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 0x7f20ac071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20ab7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20ac065d20 0x7f20ac071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20abfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 0x7f20ac071ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55224/0 (socket says 172.21.15.148:55224) 2023-10-30T12:39:49.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20abfff640 1 -- 172.21.15.148:0/3713519726 learned_addr learned my addr 172.21.15.148:0/3713519726 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:49.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20abfff640 10 monclient(hunting): get_auth_request con 0x7f20ac0afda0 auth_method 0 2023-10-30T12:39:49.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20abfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.223+0000 7f20abfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:49.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.224+0000 7f20ab7fe640 10 monclient(hunting): get_auth_request con 0x7f20ac065d20 auth_method 0 2023-10-30T12:39:49.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.224+0000 7f20ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.224+0000 7f20ab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.224+0000 7f20ab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:49.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.224+0000 7f20abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.224+0000 7f20abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.224+0000 7f20abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:49.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.224+0000 7f20ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.224+0000 7f20ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.224+0000 7f20ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:49.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.226+0000 7f20abfff640 10 monclient(hunting): handle_auth_done global_id 16266 payload 274 2023-10-30T12:39:49.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.226+0000 7f20abfff640 1 -- 172.21.15.148:0/3713519726 >> v2:172.21.15.150:3300/0 conn(0x7f20ac065d20 msgr2=0x7f20ac071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:49.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.226+0000 7f20abfff640 1 --2- 172.21.15.148:0/3713519726 >> v2:172.21.15.150:3300/0 conn(0x7f20ac065d20 0x7f20ac071580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.226+0000 7f20abfff640 1 -- 172.21.15.148:0/3713519726 >> v1:172.21.15.150:6789/0 conn(0x7f20ac06d9d0 legacy=0x7f20ac144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.226+0000 7f20abfff640 10 monclient: _finish_hunting 0 2023-10-30T12:39:49.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.226+0000 7f20abfff640 1 monclient: found mon.noname-a 2023-10-30T12:39:49.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.226+0000 7f20abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:49.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.226+0000 7f20abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:39:49.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.226+0000 7f20abfff640 1 -- 172.21.15.148:0/3713519726 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac06ec60 con 0x7f20ac0afda0 2023-10-30T12:39:49.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.227+0000 7f20abfff640 1 -- 172.21.15.148:0/3713519726 >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 msgr2=0x7f20ac071ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:39:49.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.227+0000 7f20abfff640 1 -- 172.21.15.148:0/3713519726 >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 msgr2=0x7f20ac071ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:39:49.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.227+0000 7f20abfff640 1 --2- 172.21.15.148:0/3713519726 >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 0x7f20ac071ac0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f20a4009870 tx=0x7f20a402b910 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:39:49.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.227+0000 7f20abfff640 1 --2- 172.21.15.148:0/3713519726 >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 0x7f20ac071ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:39:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20abfff640 1 --2- 172.21.15.148:0/3713519726 >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 0x7f20ac071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20abfff640 10 monclient: get_auth_request con 0x7f20ac0afda0 auth_method 0 2023-10-30T12:39:49.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20abfff640 0 --2- 172.21.15.148:0/3713519726 >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 0x7f20ac071ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-30T12:39:49.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20abfff640 1 --2- 172.21.15.148:0/3713519726 >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 0x7f20ac071ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 10 monclient: ms_handle_reset current mon v2:172.21.15.148:3300/0 2023-10-30T12:39:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 1 -- 172.21.15.148:0/3713519726 >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 msgr2=0x7f20ac071ac0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2023-10-30T12:39:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 1 --2- 172.21.15.148:0/3713519726 >> v2:172.21.15.148:3300/0 conn(0x7f20ac0afda0 0x7f20ac071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 10 monclient: _add_conns ranks=[0,2] 2023-10-30T12:39:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2098003500 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 1 --2- 172.21.15.148:0/3713519726 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2098007430 0x7f20980098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f2098007430 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:49.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 1 -- 172.21.15.148:0/3713519726 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209800a1e0 con 0x7f2098003500 2023-10-30T12:39:49.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 1 -- 172.21.15.148:0/3713519726 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209800b3e0 con 0x7f2098007430 2023-10-30T12:39:49.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:49.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.428+0000 7f20aa7fc640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.148:3300/0 2023-10-30T12:39:49.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.429+0000 7f20aaffd640 1 --2- 172.21.15.148:0/3713519726 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2098007430 0x7f20980098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.429+0000 7f20aaffd640 10 monclient(hunting): get_auth_request con 0x7f2098007430 auth_method 0 2023-10-30T12:39:49.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.429+0000 7f20aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.429+0000 7f20aaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.429+0000 7f20aaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:39:49.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.430+0000 7f20aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.430+0000 7f20aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.430+0000 7f20aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:39:49.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.432+0000 7f20aaffd640 10 monclient(hunting): handle_auth_done global_id 16266 payload 293 2023-10-30T12:39:49.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.432+0000 7f20aaffd640 1 -- 172.21.15.148:0/3713519726 >> v1:172.21.15.148:6789/0 conn(0x7f2098003500 legacy=0x7f2098005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.432+0000 7f20aaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:39:49.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.432+0000 7f20aaffd640 1 monclient: found mon.noname-b 2023-10-30T12:39:49.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.432+0000 7f20aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:49.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.432+0000 7f20aaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:49.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.432+0000 7f20aaffd640 1 -- 172.21.15.148:0/3713519726 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209800b670 con 0x7f2098007430 2023-10-30T12:39:49.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.434+0000 7f20aaffd640 1 --2- 172.21.15.148:0/3713519726 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2098007430 0x7f20980098f0 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f20940016a0 tx=0x7f20a4009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b8ec45265bc4b73 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:49.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 1 -- 172.21.15.148:0/3713519726 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20940050d0 con 0x7f2098007430 2023-10-30T12:39:49.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:49.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:49.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 10 monclient: dump: 2023-10-30T12:39:49.440 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:49.440 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:49.440 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.440 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.441 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:49.441 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:49.441 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:49.441 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:49.441 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:49.442 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 10 monclient: _finish_auth 0 2023-10-30T12:39:49.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:39:49.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:49.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 1 -- 172.21.15.148:0/3713519726 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209400e040 con 0x7f2098007430 2023-10-30T12:39:49.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:49.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 1 -- 172.21.15.148:0/3713519726 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209400c760 con 0x7f2098007430 2023-10-30T12:39:49.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20b1d1d640 5 monclient: authenticate success, global_id 16266 2023-10-30T12:39:49.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20b1d1d640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:49.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20b1d1d640 4 set_mon_vals no callback set 2023-10-30T12:39:49.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:49.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:49.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.437+0000 7f20aa7fc640 10 monclient: dump: 2023-10-30T12:39:49.444 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:49.444 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:49.444 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.445 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.445 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:49.445 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:49.445 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:49.445 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:49.446 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:49.446 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.438+0000 7f20b1d1d640 10 monclient: shutdown 2023-10-30T12:39:49.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.438+0000 7f20b1d1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:49.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.438+0000 7f20b1d1d640 1 -- 172.21.15.148:0/3713519726 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2098007430 msgr2=0x7f20980098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.438+0000 7f20b1d1d640 1 --2- 172.21.15.148:0/3713519726 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2098007430 0x7f20980098f0 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f20940016a0 tx=0x7f20a4009240 comp rx=0 tx=0).stop 2023-10-30T12:39:49.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.438+0000 7f20abfff640 1 -- 172.21.15.148:0/3713519726 reap_dead start 2023-10-30T12:39:49.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.438+0000 7f20b1d1d640 1 -- 172.21.15.148:0/3713519726 shutdown_connections 2023-10-30T12:39:49.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.438+0000 7f20b1d1d640 1 -- 172.21.15.148:0/3713519726 >> 172.21.15.148:0/3713519726 conn(0x7f20ac06a1e0 msgr2=0x7f20ac06a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:49.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.438+0000 7f20b1d1d640 1 -- 172.21.15.148:0/3713519726 shutdown_connections 2023-10-30T12:39:49.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.438+0000 7f20b1d1d640 1 -- 172.21.15.148:0/3713519726 wait complete. 2023-10-30T12:39:49.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.439+0000 7f20b1d1d640 10 monclient: build_initial_monmap 2023-10-30T12:39:49.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.439+0000 7f20b1d1d640 10 monclient: monmap: 2023-10-30T12:39:49.452 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:49.452 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:49.452 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:49.452 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:49.453 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:49.453 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:49.453 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:49.453 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:49.453 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:49.454 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.439+0000 7f20b1d1d640 1 librados: starting msgr at 2023-10-30T12:39:49.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.439+0000 7f20b1d1d640 1 librados: starting objecter 2023-10-30T12:39:49.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.439+0000 7f20b1d1d640 1 Processor -- start 2023-10-30T12:39:49.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.439+0000 7f20b1d1d640 1 -- start start 2023-10-30T12:39:49.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.439+0000 7f20b1d1d640 1 librados: setting wanted keys 2023-10-30T12:39:49.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.439+0000 7f20b1d1d640 1 librados: calling monclient init 2023-10-30T12:39:49.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.439+0000 7f20b1d1d640 10 monclient: init 2023-10-30T12:39:49.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:49.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:39:49.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2098007430 0x7f20ac140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f2098007430 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:49.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20ac13fd40 0x7f20ac1d6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f20ac13fd40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:49.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f20ac1d6a50 0x7f20ac13abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f20ac1d6a50 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac0af8f0 con 0x7f2098007430 2023-10-30T12:39:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac141400 con 0x7f20ac13fd40 2023-10-30T12:39:49.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac13bc90 con 0x7f20ac1d6a50 2023-10-30T12:39:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20b1d1d640 10 monclient(hunting): authenticate will time out at 2881.645376s 2023-10-30T12:39:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20abfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f20ac1d6a50 0x7f20ac13abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.440+0000 7f20ab7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2098007430 0x7f20ac140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20aaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20ac13fd40 0x7f20ac1d6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20abfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f20ac1d6a50 0x7f20ac13abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54036/0 (socket says 172.21.15.148:54036) 2023-10-30T12:39:49.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20abfff640 1 -- 172.21.15.148:0/2621013343 learned_addr learned my addr 172.21.15.148:0/2621013343 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:49.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20abfff640 10 monclient(hunting): get_auth_request con 0x7f20ac1d6a50 auth_method 0 2023-10-30T12:39:49.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20abfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20abfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:49.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20ab7fe640 10 monclient(hunting): get_auth_request con 0x7f2098007430 auth_method 0 2023-10-30T12:39:49.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20ab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20ab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:49.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20aaffd640 10 monclient(hunting): get_auth_request con 0x7f20ac13fd40 auth_method 0 2023-10-30T12:39:49.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20aaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20aaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:49.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:49.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.441+0000 7f20ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:49.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.442+0000 7f20aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.442+0000 7f20aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.442+0000 7f20aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:49.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20abfff640 10 monclient(hunting): handle_auth_done global_id 17717 payload 824 2023-10-30T12:39:49.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20abfff640 1 -- 172.21.15.148:0/2621013343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20ac13fd40 msgr2=0x7f20ac1d6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20abfff640 1 --2- 172.21.15.148:0/2621013343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20ac13fd40 0x7f20ac1d6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20abfff640 1 -- 172.21.15.148:0/2621013343 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2098007430 msgr2=0x7f20ac140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20abfff640 1 --2- 172.21.15.148:0/2621013343 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2098007430 0x7f20ac140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20abfff640 10 monclient: _finish_hunting 0 2023-10-30T12:39:49.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20abfff640 1 monclient: found mon.noname-c 2023-10-30T12:39:49.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:49.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20abfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:39:49.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20abfff640 1 -- 172.21.15.148:0/2621013343 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac1d6eb0 con 0x7f20ac1d6a50 2023-10-30T12:39:49.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.444+0000 7f20ab7fe640 1 --2- 172.21.15.148:0/2621013343 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2098007430 0x7f20ac140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:39:49.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.446+0000 7f20abfff640 1 --2- 172.21.15.148:0/2621013343 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f20ac1d6a50 0x7f20ac13abc0 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7f20a4009960 tx=0x7f20a4032670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:49.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.446+0000 7f20a3fff640 10 client.?.objecter ms_handle_connect 0x7f20ac1d6a50 2023-10-30T12:39:49.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.446+0000 7f20a3fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:49.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 1 -- 172.21.15.148:0/2621013343 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4004020 con 0x7f20ac1d6a50 2023-10-30T12:39:49.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:49.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:49.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 10 monclient: dump: 2023-10-30T12:39:49.467 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:49.468 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:49.468 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.468 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.468 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:49.468 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:49.469 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:49.469 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:49.469 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:49.469 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 10 monclient: _finish_auth 0 2023-10-30T12:39:49.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:39:49.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:49.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 1 -- 172.21.15.148:0/2621013343 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a4004190 con 0x7f20ac1d6a50 2023-10-30T12:39:49.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:49.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 1 -- 172.21.15.148:0/2621013343 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a40329a0 con 0x7f20ac1d6a50 2023-10-30T12:39:49.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20b1d1d640 5 monclient: authenticate success, global_id 17717 2023-10-30T12:39:49.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20b1d1d640 10 monclient: _renew_subs 2023-10-30T12:39:49.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20b1d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:49.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac13f770 con 0x7f20ac1d6a50 2023-10-30T12:39:49.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a9ffb640 4 set_mon_vals no callback set 2023-10-30T12:39:49.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:49.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:49.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20a3fff640 10 monclient: dump: 2023-10-30T12:39:49.472 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:49.472 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:49.472 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.472 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.473 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:49.473 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:49.473 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:49.473 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:49.473 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:49.474 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20b1d1d640 10 client.17717.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:49.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20b1d1d640 10 monclient: _renew_subs 2023-10-30T12:39:49.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20b1d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:49.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac141170 con 0x7f20ac1d6a50 2023-10-30T12:39:49.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.450+0000 7f20b1d1d640 1 librados: init done 2023-10-30T12:39:49.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.451+0000 7f20b1d1d640 10 client.17717.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:49.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.451+0000 7f20b1d1d640 15 client.17717.objecter _session_command_op_assign -1 1 2023-10-30T12:39:49.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.451+0000 7f20b1d1d640 20 client.17717.objecter _get_session osd=-1 returning homeless 2023-10-30T12:39:49.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.451+0000 7f20b1d1d640 10 client.17717.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:49.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.451+0000 7f20b1d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:49.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.451+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f20ac075550 con 0x7f20ac1d6a50 2023-10-30T12:39:49.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.470+0000 7f20a3fff640 1 -- 172.21.15.148:0/2621013343 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f20a4032b10 con 0x7f20ac1d6a50 2023-10-30T12:39:49.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.470+0000 7f20a3fff640 1 --2- 172.21.15.148:0/2621013343 >> v2:172.21.15.148:6800/3205090179 conn(0x7f208c06c7c0 0x7f208c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.470+0000 7f20ab7fe640 1 --2- 172.21.15.148:0/2621013343 >> v2:172.21.15.148:6800/3205090179 conn(0x7f208c06c7c0 0x7f208c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.470+0000 7f20ab7fe640 10 monclient: get_auth_request con 0x7f208c06c7c0 auth_method 0 2023-10-30T12:39:49.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.470+0000 7f20ab7fe640 1 --2- 172.21.15.148:0/2621013343 >> v2:172.21.15.148:6800/3205090179 conn(0x7f208c06c7c0 0x7f208c06ec80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f209c009a40 tx=0x7f209c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:49.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.471+0000 7f20a3fff640 10 client.17717.objecter ms_handle_connect 0x7f208c06c7c0 2023-10-30T12:39:49.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 1 -- 172.21.15.148:0/2621013343 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f20a40795b0 con 0x7f20ac1d6a50 2023-10-30T12:39:49.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 10 client.17717.objecter ms_dispatch 0x7f20ac000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:49.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 3 client.17717.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:49.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 3 client.17717.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:49.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 10 client.17717.objecter checking command 1 2023-10-30T12:39:49.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 1 --2- 172.21.15.148:0/2621013343 >> v2:172.21.15.148:6801/2368806934 conn(0x7f208c073ae0 0x7f208c075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 20 client.17717.objecter _get_session s=0x7f20a40329a0 osd=2 3 2023-10-30T12:39:49.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 20 client.17717.objecter put_session s=0x7f20a40329a0 osd=2 3 2023-10-30T12:39:49.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 15 client.17717.objecter _session_command_op_remove -1 1 2023-10-30T12:39:49.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 20 client.17717.objecter _get_session s=0x7f20a40329a0 osd=2 3 2023-10-30T12:39:49.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 20 client.17717.objecter get_session s=0x7f20a40329a0 osd=2 3 2023-10-30T12:39:49.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 15 client.17717.objecter _session_command_op_assign 2 1 2023-10-30T12:39:49.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 20 client.17717.objecter put_session s=0x7f20a40329a0 osd=2 4 2023-10-30T12:39:49.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 10 client.17717.objecter _send_command 1 2023-10-30T12:39:49.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 1 -- 172.21.15.148:0/2621013343 --> v2:172.21.15.148:6801/2368806934 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20a4004020 con 0x7f208c073ae0 2023-10-30T12:39:49.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.472+0000 7f20a3fff640 20 client.17717.objecter dump_active .. 0 homeless 2023-10-30T12:39:49.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.473+0000 7f20aaffd640 1 --2- 172.21.15.148:0/2621013343 >> v2:172.21.15.148:6801/2368806934 conn(0x7f208c073ae0 0x7f208c075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.473+0000 7f20aaffd640 10 monclient: get_auth_request con 0x7f208c073ae0 auth_method 0 2023-10-30T12:39:49.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.473+0000 7f20a3fff640 1 -- 172.21.15.148:0/2621013343 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20a4037070 con 0x7f20ac1d6a50 2023-10-30T12:39:49.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.473+0000 7f20a3fff640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:39:49.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.475+0000 7f20a3fff640 10 client.17717.objecter ms_handle_connect 0x7f208c073ae0 2023-10-30T12:39:49.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.475+0000 7f20aaffd640 1 --2- 172.21.15.148:0/2621013343 >> v2:172.21.15.148:6801/2368806934 conn(0x7f208c073ae0 0x7f208c075f80 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:49.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.499+0000 7f20a3fff640 1 -- 172.21.15.148:0/2621013343 <== osd.2 v2:172.21.15.148:6801/2368806934 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f20a4004020 con 0x7f208c073ae0 2023-10-30T12:39:49.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.499+0000 7f20a3fff640 10 client.17717.objecter ms_dispatch 0x7f20ac000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:39:49.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.499+0000 7f20a3fff640 10 client.17717.objecter _finish_command 1 = system:0 2023-10-30T12:39:49.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.499+0000 7f20a3fff640 20 client.17717.objecter put_session s=0x7f20a40329a0 osd=2 4 2023-10-30T12:39:49.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.499+0000 7f20a3fff640 15 client.17717.objecter _session_command_op_remove 2 1 2023-10-30T12:39:49.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 10 client.17717.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:39:49.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 15 client.17717.objecter _session_command_op_assign -1 2 2023-10-30T12:39:49.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 20 client.17717.objecter _get_session s=0x7f20a40329a0 osd=2 3 2023-10-30T12:39:49.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 20 client.17717.objecter put_session s=0x7f20a40329a0 osd=2 3 2023-10-30T12:39:49.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 20 client.17717.objecter _get_session s=0x7f20a40329a0 osd=2 3 2023-10-30T12:39:49.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 15 client.17717.objecter _session_command_op_remove -1 2 2023-10-30T12:39:49.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 20 client.17717.objecter get_session s=0x7f20a40329a0 osd=2 3 2023-10-30T12:39:49.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 15 client.17717.objecter _session_command_op_assign 2 2 2023-10-30T12:39:49.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 20 client.17717.objecter put_session s=0x7f20a40329a0 osd=2 4 2023-10-30T12:39:49.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 10 client.17717.objecter _send_command 2 2023-10-30T12:39:49.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.511+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 --> v2:172.21.15.148:6801/2368806934 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f20ac1d6eb0 con 0x7f208c073ae0 2023-10-30T12:39:49.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20a3fff640 1 -- 172.21.15.148:0/2621013343 <== osd.2 v2:172.21.15.148:6801/2368806934 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f20ac1d6eb0 con 0x7f208c073ae0 2023-10-30T12:39:49.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20a3fff640 10 client.17717.objecter ms_dispatch 0x7f20ac000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:39:49.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20a3fff640 10 client.17717.objecter _finish_command 2 = system:0 2023-10-30T12:39:49.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20a3fff640 20 client.17717.objecter put_session s=0x7f20a40329a0 osd=2 4 2023-10-30T12:39:49.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20a3fff640 15 client.17717.objecter _session_command_op_remove 2 2 2023-10-30T12:39:49.527 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T12:39:49.528 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:39:49.528 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T12:39:49.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 10 librados: watch_flush enter 2023-10-30T12:39:49.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 10 librados: watch_flush exit 2023-10-30T12:39:49.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 10 client.17717.objecter close_session for osd.2 2023-10-30T12:39:49.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 >> v2:172.21.15.148:6801/2368806934 conn(0x7f208c073ae0 msgr2=0x7f208c075f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 --2- 172.21.15.148:0/2621013343 >> v2:172.21.15.148:6801/2368806934 conn(0x7f208c073ae0 0x7f208c075f80 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 20 client.17717.objecter put_session s=0x7f20a40329a0 osd=2 1 2023-10-30T12:39:49.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 20 client.17717.objecter shutdown clearing up homeless session... 2023-10-30T12:39:49.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 10 client.17717.objecter successfully canceled tick 2023-10-30T12:39:49.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 >> v2:172.21.15.148:6800/3205090179 conn(0x7f208c06c7c0 msgr2=0x7f208c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 --2- 172.21.15.148:0/2621013343 >> v2:172.21.15.148:6800/3205090179 conn(0x7f208c06c7c0 0x7f208c06ec80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f209c009a40 tx=0x7f209c00b040 comp rx=0 tx=0).stop 2023-10-30T12:39:49.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 10 monclient: shutdown 2023-10-30T12:39:49.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:49.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f20ac1d6a50 msgr2=0x7f20ac13abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 --2- 172.21.15.148:0/2621013343 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f20ac1d6a50 0x7f20ac13abc0 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7f20a4009960 tx=0x7f20a4032670 comp rx=0 tx=0).stop 2023-10-30T12:39:49.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20abfff640 1 -- 172.21.15.148:0/2621013343 reap_dead start 2023-10-30T12:39:49.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 shutdown_connections 2023-10-30T12:39:49.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 >> 172.21.15.148:0/2621013343 conn(0x7f20ac06a1e0 msgr2=0x7f20ac0b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:49.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 shutdown_connections 2023-10-30T12:39:49.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 -- 172.21.15.148:0/2621013343 wait complete. 2023-10-30T12:39:49.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.525+0000 7f20b1d1d640 1 librados: shutdown 2023-10-30T12:39:49.538 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T12:39:49.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.636+0000 7ff8defa5640 10 monclient: get_monmap_and_config 2023-10-30T12:39:49.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.636+0000 7ff8defa5640 10 monclient: build_initial_monmap 2023-10-30T12:39:49.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.636+0000 7ff8defa5640 10 monclient: monmap: 2023-10-30T12:39:49.638 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:49.639 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:49.639 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:49.637267+0000 2023-10-30T12:39:49.639 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:49.637267+0000 2023-10-30T12:39:49.639 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:49.639 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:49.639 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:49.640 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:49.640 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:49.640 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:49.640 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:49.640 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.636+0000 7ff8defa5640 1 Processor -- start 2023-10-30T12:39:49.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 1 -- start start 2023-10-30T12:39:49.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 10 monclient: init 2023-10-30T12:39:49.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:49.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-10-30T12:39:49.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8d80a9210 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:49.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff8d8071340 0x7ff8d80aece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 10 monclient(hunting): picked mon.noname-c con 0x7ff8d8071340 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:49.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff8d80af220 0x7ff8d80a8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 10 monclient(hunting): picked mon.noname-a con 0x7ff8d80af220 addr v2:172.21.15.148:3300/0 2023-10-30T12:39:49.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d80aa630 con 0x7ff8d80a9210 2023-10-30T12:39:49.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d8067340 con 0x7ff8d80af220 2023-10-30T12:39:49.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d80aa390 con 0x7ff8d8071340 2023-10-30T12:39:49.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:49.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.637+0000 7ff8defa5640 10 monclient(hunting): authenticate will time out at 2881.842480s 2023-10-30T12:39:49.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8dd542640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff8d80af220 0x7ff8d80a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8d7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff8d8071340 0x7ff8d80aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8dd542640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff8d80af220 0x7ff8d80a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55252/0 (socket says 172.21.15.148:55252) 2023-10-30T12:39:49.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8dd542640 1 -- 172.21.15.148:0/2030420797 learned_addr learned my addr 172.21.15.148:0/2030420797 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:49.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8dd542640 10 monclient(hunting): get_auth_request con 0x7ff8d80af220 auth_method 0 2023-10-30T12:39:49.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8dd542640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8dd542640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8dd542640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:49.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8d7fff640 10 monclient(hunting): get_auth_request con 0x7ff8d8071340 auth_method 0 2023-10-30T12:39:49.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8d7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.638+0000 7ff8d7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:49.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.639+0000 7ff8dd542640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.639+0000 7ff8dd542640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.639+0000 7ff8dd542640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:49.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.639+0000 7ff8d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.639+0000 7ff8d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.639+0000 7ff8d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:49.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.640+0000 7ff8d7fff640 10 monclient(hunting): handle_auth_done global_id 22090 payload 274 2023-10-30T12:39:49.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.640+0000 7ff8d7fff640 1 -- 172.21.15.148:0/2030420797 >> v2:172.21.15.148:3300/0 conn(0x7ff8d80af220 msgr2=0x7ff8d80a8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:49.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.640+0000 7ff8d7fff640 1 --2- 172.21.15.148:0/2030420797 >> v2:172.21.15.148:3300/0 conn(0x7ff8d80af220 0x7ff8d80a8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.640+0000 7ff8d7fff640 1 -- 172.21.15.148:0/2030420797 >> v1:172.21.15.148:6789/0 conn(0x7ff8d80a9210 legacy=0x7ff8d813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:49.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.640+0000 7ff8d7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:39:49.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.640+0000 7ff8d7fff640 1 monclient: found mon.noname-c 2023-10-30T12:39:49.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.640+0000 7ff8d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:49.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.640+0000 7ff8d7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:39:49.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.640+0000 7ff8d7fff640 1 -- 172.21.15.148:0/2030420797 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d80af670 con 0x7ff8d8071340 2023-10-30T12:39:49.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.642+0000 7ff8d7fff640 1 --2- 172.21.15.148:0/2030420797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d8071340 0x7ff8d80aece0 secure :-1 s=READY pgs=5441 cs=0 l=1 rev1=1 crypto rx=0x7ff8d806bfc0 tx=0x7ff8cc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eac7d2041e54504 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:49.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 1 -- 172.21.15.148:0/2030420797 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc0315a0 con 0x7ff8d8071340 2023-10-30T12:39:49.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:49.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:49.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 10 monclient: dump: 2023-10-30T12:39:49.655 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:49.656 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:49.656 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.656 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.656 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:49.656 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:49.657 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:49.657 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:49.657 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:49.657 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 10 monclient: _finish_auth 0 2023-10-30T12:39:49.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:39:49.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:49.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 1 -- 172.21.15.148:0/2030420797 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc031710 con 0x7ff8d8071340 2023-10-30T12:39:49.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:49.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 1 -- 172.21.15.148:0/2030420797 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc031ab0 con 0x7ff8d8071340 2023-10-30T12:39:49.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 5 monclient: authenticate success, global_id 22090 2023-10-30T12:39:49.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:49.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 4 set_mon_vals no callback set 2023-10-30T12:39:49.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:49.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:49.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8d77fe640 10 monclient: dump: 2023-10-30T12:39:49.660 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:49.660 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:49.660 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.660 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.661 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:49.661 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:49.661 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:49.661 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:49.661 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:49.662 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 10 monclient: shutdown 2023-10-30T12:39:49.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:49.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 1 -- 172.21.15.148:0/2030420797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d8071340 msgr2=0x7ff8d80aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 1 --2- 172.21.15.148:0/2030420797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d8071340 0x7ff8d80aece0 secure :-1 s=READY pgs=5441 cs=0 l=1 rev1=1 crypto rx=0x7ff8d806bfc0 tx=0x7ff8cc02b840 comp rx=0 tx=0).stop 2023-10-30T12:39:49.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 1 -- 172.21.15.148:0/2030420797 shutdown_connections 2023-10-30T12:39:49.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 1 --2- 172.21.15.148:0/2030420797 >> v2:172.21.15.148:3300/0 conn(0x7ff8d80af220 0x7ff8d80a8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 1 --2- 172.21.15.148:0/2030420797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d8071340 0x7ff8d80aece0 unknown :-1 s=CLOSED pgs=5441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.646+0000 7ff8defa5640 1 -- 172.21.15.148:0/2030420797 >> 172.21.15.148:0/2030420797 conn(0x7ff8d80b3260 msgr2=0x7ff8d80b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:49.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.647+0000 7ff8defa5640 1 -- 172.21.15.148:0/2030420797 shutdown_connections 2023-10-30T12:39:49.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.647+0000 7ff8defa5640 1 -- 172.21.15.148:0/2030420797 wait complete. 2023-10-30T12:39:49.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.647+0000 7ff8defa5640 10 monclient: build_initial_monmap 2023-10-30T12:39:49.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.647+0000 7ff8defa5640 10 monclient: monmap: 2023-10-30T12:39:49.664 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:49.665 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:49.665 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:49.665 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:49.665 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:49.665 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:49.666 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:49.666 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:49.666 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:49.666 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.647+0000 7ff8defa5640 1 librados: starting msgr at 2023-10-30T12:39:49.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.647+0000 7ff8defa5640 1 librados: starting objecter 2023-10-30T12:39:49.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.647+0000 7ff8defa5640 1 Processor -- start 2023-10-30T12:39:49.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 1 -- start start 2023-10-30T12:39:49.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 1 librados: setting wanted keys 2023-10-30T12:39:49.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 1 librados: calling monclient init 2023-10-30T12:39:49.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient: init 2023-10-30T12:39:49.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:49.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:39:49.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff8d8071340 0x7ff8d813d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient(hunting): picked mon.noname-c con 0x7ff8d8071340 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:49.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d80a9210 0x7ff8d81d6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient(hunting): picked mon.noname-a con 0x7ff8d80a9210 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:49.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d80af220 0x7ff8d8140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient(hunting): picked mon.noname-b con 0x7ff8d80af220 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:49.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d80ab840 con 0x7ff8d80a9210 2023-10-30T12:39:49.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d81d5e20 con 0x7ff8d80af220 2023-10-30T12:39:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d81d6100 con 0x7ff8d8071340 2023-10-30T12:39:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:49.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.648+0000 7ff8defa5640 10 monclient(hunting): authenticate will time out at 2881.853568s 2023-10-30T12:39:49.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8d7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d80a9210 0x7ff8d81d6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8dcd41640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff8d8071340 0x7ff8d813d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8dd542640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d80af220 0x7ff8d8140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8d7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d80a9210 0x7ff8d81d6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55262/0 (socket says 172.21.15.148:55262) 2023-10-30T12:39:49.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8d7fff640 1 -- 172.21.15.148:0/2142051619 learned_addr learned my addr 172.21.15.148:0/2142051619 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:49.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8d7fff640 10 monclient(hunting): get_auth_request con 0x7ff8d80a9210 auth_method 0 2023-10-30T12:39:49.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8d7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8d7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:49.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8dcd41640 10 monclient(hunting): get_auth_request con 0x7ff8d8071340 auth_method 0 2023-10-30T12:39:49.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8dcd41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8dcd41640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8dcd41640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:49.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8dd542640 10 monclient(hunting): get_auth_request con 0x7ff8d80af220 auth_method 0 2023-10-30T12:39:49.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8dd542640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:49.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8dd542640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:49.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8dd542640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:49.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.649+0000 7ff8d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:49.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.650+0000 7ff8dd542640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.650+0000 7ff8dd542640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.650+0000 7ff8dd542640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:49.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.650+0000 7ff8dcd41640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:49.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.650+0000 7ff8dcd41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:49.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.650+0000 7ff8dcd41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:49.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.652+0000 7ff8dd542640 10 monclient(hunting): handle_auth_done global_id 22093 payload 824 2023-10-30T12:39:49.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.652+0000 7ff8dd542640 1 -- 172.21.15.148:0/2142051619 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff8d8071340 msgr2=0x7ff8d813d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.652+0000 7ff8dd542640 1 --2- 172.21.15.148:0/2142051619 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff8d8071340 0x7ff8d813d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.652+0000 7ff8dd542640 1 -- 172.21.15.148:0/2142051619 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d80a9210 msgr2=0x7ff8d81d6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:49.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.652+0000 7ff8dd542640 1 --2- 172.21.15.148:0/2142051619 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d80a9210 0x7ff8d81d6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:49.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.652+0000 7ff8dd542640 10 monclient: _finish_hunting 0 2023-10-30T12:39:49.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.652+0000 7ff8dd542640 1 monclient: found mon.noname-b 2023-10-30T12:39:49.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.652+0000 7ff8dd542640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:49.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.652+0000 7ff8dd542640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:39:49.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.652+0000 7ff8dd542640 1 -- 172.21.15.148:0/2142051619 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d813f560 con 0x7ff8d80af220 2023-10-30T12:39:49.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.654+0000 7ff8dd542640 1 --2- 172.21.15.148:0/2142051619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d80af220 0x7ff8d8140ed0 secure :-1 s=READY pgs=5442 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0010de0 tx=0x7ff8d000e490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:49.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.654+0000 7ff8d57fa640 10 client.?.objecter ms_handle_connect 0x7ff8d80af220 2023-10-30T12:39:49.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.654+0000 7ff8d57fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:49.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 1 -- 172.21.15.148:0/2142051619 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0011040 con 0x7ff8d80af220 2023-10-30T12:39:49.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:49.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:49.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 10 monclient: dump: 2023-10-30T12:39:49.680 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:49.680 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:49.681 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.681 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.681 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:49.681 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:49.681 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:49.682 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:49.682 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:49.682 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 10 monclient: _finish_auth 0 2023-10-30T12:39:49.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 10 monclient: _check_auth_tickets 2023-10-30T12:39:49.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:49.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 1 -- 172.21.15.148:0/2142051619 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d000eb20 con 0x7ff8d80af220 2023-10-30T12:39:49.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:49.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 1 -- 172.21.15.148:0/2142051619 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d000eec0 con 0x7ff8d80af220 2023-10-30T12:39:49.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8defa5640 5 monclient: authenticate success, global_id 22093 2023-10-30T12:39:49.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8defa5640 10 monclient: _renew_subs 2023-10-30T12:39:49.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8defa5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:49.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8defa5640 1 -- 172.21.15.148:0/2142051619 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d81408f0 con 0x7ff8d80af220 2023-10-30T12:39:49.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:49.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:49.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d57fa640 10 monclient: dump: 2023-10-30T12:39:49.685 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:49.685 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:49.685 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.685 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:49.685 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:49.685 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:49.686 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:49.686 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:49.686 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:49.686 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:49.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8d6ffd640 4 set_mon_vals no callback set 2023-10-30T12:39:49.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8defa5640 10 client.22093.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:49.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8defa5640 10 monclient: _renew_subs 2023-10-30T12:39:49.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8defa5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:49.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8defa5640 1 -- 172.21.15.148:0/2142051619 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d81d6d40 con 0x7ff8d80af220 2023-10-30T12:39:49.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.658+0000 7ff8defa5640 1 librados: init done 2023-10-30T12:39:49.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.659+0000 7ff8defa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:49.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.659+0000 7ff8defa5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:49.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.659+0000 7ff8defa5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:49.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.659+0000 7ff8defa5640 1 -- 172.21.15.148:0/2142051619 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d80ab3a0 con 0x7ff8d80af220 2023-10-30T12:39:49.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.676+0000 7ff8d57fa640 1 -- 172.21.15.148:0/2142051619 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7ff8d0003990 con 0x7ff8d80af220 2023-10-30T12:39:49.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.677+0000 7ff8d57fa640 1 --2- 172.21.15.148:0/2142051619 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff8c4070c20 0x7ff8c40730e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:49.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.677+0000 7ff8dcd41640 1 --2- 172.21.15.148:0/2142051619 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff8c4070c20 0x7ff8c40730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:49.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.677+0000 7ff8dcd41640 10 monclient: get_auth_request con 0x7ff8c4070c20 auth_method 0 2023-10-30T12:39:49.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.677+0000 7ff8dcd41640 1 --2- 172.21.15.148:0/2142051619 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff8c4070c20 0x7ff8c40730e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80017f0 tx=0x7ff8c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:49.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.677+0000 7ff8d57fa640 10 client.22093.objecter ms_handle_connect 0x7ff8c4070c20 2023-10-30T12:39:49.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.678+0000 7ff8d57fa640 1 -- 172.21.15.148:0/2142051619 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7ff8d0058e90 con 0x7ff8d80af220 2023-10-30T12:39:49.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.678+0000 7ff8d57fa640 10 client.22093.objecter ms_dispatch 0x7ff8d8000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:49.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.678+0000 7ff8d57fa640 3 client.22093.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:49.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.678+0000 7ff8d57fa640 3 client.22093.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:49.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.678+0000 7ff8d57fa640 20 client.22093.objecter dump_active .. 0 homeless 2023-10-30T12:39:49.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.913+0000 7ff8d57fa640 1 -- 172.21.15.148:0/2142051619 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff8d005cf30 con 0x7ff8d80af220 2023-10-30T12:39:49.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.913+0000 7ff8d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:49.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:49.913+0000 7ff8d57fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:50.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.082+0000 7ff8defa5640 1 -- 172.21.15.148:0/2142051619 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8d81d5f70 con 0x7ff8c4070c20 2023-10-30T12:39:50.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.083+0000 7ff8d57fa640 1 -- 172.21.15.148:0/2142051619 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff8d81d5f70 con 0x7ff8c4070c20 2023-10-30T12:39:50.085 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T12:39:50.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.084+0000 7ff8defa5640 10 librados: watch_flush enter 2023-10-30T12:39:50.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.084+0000 7ff8defa5640 10 librados: watch_flush exit 2023-10-30T12:39:50.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.084+0000 7ff8defa5640 20 client.22093.objecter shutdown clearing up homeless session... 2023-10-30T12:39:50.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.084+0000 7ff8defa5640 10 client.22093.objecter successfully canceled tick 2023-10-30T12:39:50.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.084+0000 7ff8defa5640 1 -- 172.21.15.148:0/2142051619 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff8c4070c20 msgr2=0x7ff8c40730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.084+0000 7ff8defa5640 1 --2- 172.21.15.148:0/2142051619 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff8c4070c20 0x7ff8c40730e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80017f0 tx=0x7ff8c8009040 comp rx=0 tx=0).stop 2023-10-30T12:39:50.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.084+0000 7ff8defa5640 10 monclient: shutdown 2023-10-30T12:39:50.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.084+0000 7ff8defa5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:50.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.084+0000 7ff8defa5640 1 -- 172.21.15.148:0/2142051619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d80af220 msgr2=0x7ff8d8140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.084+0000 7ff8defa5640 1 --2- 172.21.15.148:0/2142051619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d80af220 0x7ff8d8140ed0 secure :-1 s=READY pgs=5442 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0010de0 tx=0x7ff8d000e490 comp rx=0 tx=0).stop 2023-10-30T12:39:50.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.085+0000 7ff8defa5640 1 -- 172.21.15.148:0/2142051619 shutdown_connections 2023-10-30T12:39:50.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.085+0000 7ff8defa5640 1 --2- 172.21.15.148:0/2142051619 >> v2:172.21.15.148:6800/3205090179 conn(0x7ff8c4070c20 0x7ff8c40730e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.085+0000 7ff8defa5640 1 --2- 172.21.15.148:0/2142051619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d80af220 0x7ff8d8140ed0 unknown :-1 s=CLOSED pgs=5442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.085+0000 7ff8defa5640 1 --2- 172.21.15.148:0/2142051619 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d80a9210 0x7ff8d81d6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.085+0000 7ff8defa5640 1 --2- 172.21.15.148:0/2142051619 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff8d8071340 0x7ff8d813d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.085+0000 7ff8defa5640 1 -- 172.21.15.148:0/2142051619 >> 172.21.15.148:0/2142051619 conn(0x7ff8d80b3260 msgr2=0x7ff8d80ac140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:50.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.085+0000 7ff8defa5640 1 -- 172.21.15.148:0/2142051619 shutdown_connections 2023-10-30T12:39:50.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.085+0000 7ff8defa5640 1 -- 172.21.15.148:0/2142051619 wait complete. 2023-10-30T12:39:50.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.085+0000 7ff8defa5640 1 librados: shutdown 2023-10-30T12:39:50.098 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T12:39:50.098 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:39:50.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.200+0000 7f06fa61f640 10 monclient: get_monmap_and_config 2023-10-30T12:39:50.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.200+0000 7f06fa61f640 10 monclient: build_initial_monmap 2023-10-30T12:39:50.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.200+0000 7f06fa61f640 10 monclient: monmap: 2023-10-30T12:39:50.202 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:50.202 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.203 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:50.201237+0000 2023-10-30T12:39:50.203 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:50.201237+0000 2023-10-30T12:39:50.203 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:50.203 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.204 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:50.204 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:50.204 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:50.204 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:50.205 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:50.205 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.200+0000 7f06fa61f640 1 Processor -- start 2023-10-30T12:39:50.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.200+0000 7f06fa61f640 1 -- start start 2023-10-30T12:39:50.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.200+0000 7f06fa61f640 10 monclient: init 2023-10-30T12:39:50.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:50.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-10-30T12:39:50.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06f40b7c10 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:50.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f4141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 10 monclient(hunting): picked mon.noname-b con 0x7f06f4141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:50.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f06f413e730 0x7f06f413eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 10 monclient(hunting): picked mon.noname-a con 0x7f06f413e730 addr v2:172.21.15.148:3300/0 2023-10-30T12:39:50.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f40a9350 con 0x7f06f40b7c10 2023-10-30T12:39:50.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:50.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f40670a0 con 0x7f06f413e730 2023-10-30T12:39:50.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:50.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f40b7ff0 con 0x7f06f4141940 2023-10-30T12:39:50.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:50.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.201+0000 7f06fa61f640 10 monclient(hunting): authenticate will time out at 2882.406475s 2023-10-30T12:39:50.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f37fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f4141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f8bbc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f06f413e730 0x7f06f413eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f37fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f4141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54066/0 (socket says 172.21.15.148:54066) 2023-10-30T12:39:50.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f37fe640 1 -- 172.21.15.148:0/1356680210 learned_addr learned my addr 172.21.15.148:0/1356680210 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:50.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f37fe640 10 monclient(hunting): get_auth_request con 0x7f06f4141940 auth_method 0 2023-10-30T12:39:50.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:50.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:50.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:50.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f8bbc640 10 monclient(hunting): get_auth_request con 0x7f06f413e730 auth_method 0 2023-10-30T12:39:50.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f8bbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:50.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f8bbc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:50.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f8bbc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:50.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:50.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:50.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:50.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f8bbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:50.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.202+0000 7f06f8bbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:50.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.203+0000 7f06f8bbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:50.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.204+0000 7f06f37fe640 10 monclient(hunting): handle_auth_done global_id 17723 payload 274 2023-10-30T12:39:50.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.204+0000 7f06f37fe640 1 -- 172.21.15.148:0/1356680210 >> v2:172.21.15.148:3300/0 conn(0x7f06f413e730 msgr2=0x7f06f413eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:50.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.204+0000 7f06f37fe640 1 --2- 172.21.15.148:0/1356680210 >> v2:172.21.15.148:3300/0 conn(0x7f06f413e730 0x7f06f413eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.204+0000 7f06f37fe640 1 -- 172.21.15.148:0/1356680210 >> v1:172.21.15.148:6789/0 conn(0x7f06f40b7c10 legacy=0x7f06f40a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:50.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.204+0000 7f06f37fe640 10 monclient: _finish_hunting 0 2023-10-30T12:39:50.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.204+0000 7f06f37fe640 1 monclient: found mon.noname-b 2023-10-30T12:39:50.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.204+0000 7f06f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:50.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.204+0000 7f06f37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:50.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.204+0000 7f06f37fe640 1 -- 172.21.15.148:0/1356680210 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f40aa6a0 con 0x7f06f4141940 2023-10-30T12:39:50.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.204+0000 7f06f8bbc640 1 --2- 172.21.15.148:0/1356680210 >> v2:172.21.15.148:3300/0 conn(0x7f06f413e730 0x7f06f413eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:39:50.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.206+0000 7f06f37fe640 1 --2- 172.21.15.148:0/1356680210 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f4141d60 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7f06dc009870 tx=0x7f06dc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d67e3d7c9a0db42 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:50.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 1 -- 172.21.15.148:0/1356680210 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc0317c0 con 0x7f06f4141940 2023-10-30T12:39:50.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:50.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:50.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 10 monclient: dump: 2023-10-30T12:39:50.222 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:50.222 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.222 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.222 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.222 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:50.223 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.223 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:50.223 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:50.223 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:50.224 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:39:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:39:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 1 -- 172.21.15.148:0/1356680210 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06dc02e070 con 0x7f06f4141940 2023-10-30T12:39:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06fa61f640 5 monclient: authenticate success, global_id 17723 2023-10-30T12:39:50.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06fa61f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:39:50.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:50.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06fa61f640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:50.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06fa61f640 4 set_mon_vals no callback set 2023-10-30T12:39:50.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 1 -- 172.21.15.148:0/1356680210 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc031b50 con 0x7f06f4141940 2023-10-30T12:39:50.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:50.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:50.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.209+0000 7f06f2ffd640 10 monclient: dump: 2023-10-30T12:39:50.226 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:50.227 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.227 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.227 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.227 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:50.227 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.227 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:50.228 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:50.228 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:50.228 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.210+0000 7f06fa61f640 10 monclient: shutdown 2023-10-30T12:39:50.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.210+0000 7f06fa61f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:50.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.210+0000 7f06fa61f640 1 -- 172.21.15.148:0/1356680210 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 msgr2=0x7f06f4141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.210+0000 7f06fa61f640 1 --2- 172.21.15.148:0/1356680210 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f4141d60 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7f06dc009870 tx=0x7f06dc02b860 comp rx=0 tx=0).stop 2023-10-30T12:39:50.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.210+0000 7f06fa61f640 1 -- 172.21.15.148:0/1356680210 shutdown_connections 2023-10-30T12:39:50.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.210+0000 7f06fa61f640 1 --2- 172.21.15.148:0/1356680210 >> v2:172.21.15.148:3300/0 conn(0x7f06f413e730 0x7f06f413eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.210+0000 7f06fa61f640 1 --2- 172.21.15.148:0/1356680210 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f4141d60 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.210+0000 7f06fa61f640 1 -- 172.21.15.148:0/1356680210 >> 172.21.15.148:0/1356680210 conn(0x7f06f406c660 msgr2=0x7f06f40b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:50.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.210+0000 7f06fa61f640 1 -- 172.21.15.148:0/1356680210 shutdown_connections 2023-10-30T12:39:50.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.210+0000 7f06fa61f640 1 -- 172.21.15.148:0/1356680210 wait complete. 2023-10-30T12:39:50.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.211+0000 7f06fa61f640 10 monclient: build_initial_monmap 2023-10-30T12:39:50.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.211+0000 7f06fa61f640 10 monclient: monmap: 2023-10-30T12:39:50.231 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:50.231 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:50.231 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:50.231 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:50.231 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:50.232 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:50.232 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:50.232 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:50.232 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:50.232 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.211+0000 7f06fa61f640 1 librados: starting msgr at 2023-10-30T12:39:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.211+0000 7f06fa61f640 1 librados: starting objecter 2023-10-30T12:39:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.211+0000 7f06fa61f640 1 Processor -- start 2023-10-30T12:39:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.211+0000 7f06fa61f640 1 -- start start 2023-10-30T12:39:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.211+0000 7f06fa61f640 1 librados: setting wanted keys 2023-10-30T12:39:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.211+0000 7f06fa61f640 1 librados: calling monclient init 2023-10-30T12:39:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.211+0000 7f06fa61f640 10 monclient: init 2023-10-30T12:39:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:39:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06f40b7c10 0x7f06f41da550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 10 monclient(hunting): picked mon.noname-b con 0x7f06f40b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06f413e730 0x7f06f40b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 10 monclient(hunting): picked mon.noname-a con 0x7f06f413e730 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f411eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 10 monclient(hunting): picked mon.noname-c con 0x7f06f4141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f413f530 con 0x7f06f413e730 2023-10-30T12:39:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f41dabe0 con 0x7f06f40b7c10 2023-10-30T12:39:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f41db100 con 0x7f06f4141940 2023-10-30T12:39:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06fa61f640 10 monclient(hunting): authenticate will time out at 2882.417440s 2023-10-30T12:39:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.212+0000 7f06f8bbc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f411eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06f413e730 0x7f06f40b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06f40b7c10 0x7f06f41da550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f8bbc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f411eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:54076/0 (socket says 172.21.15.148:54076) 2023-10-30T12:39:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f8bbc640 1 -- 172.21.15.148:0/4087602167 learned_addr learned my addr 172.21.15.148:0/4087602167 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f8bbc640 10 monclient(hunting): get_auth_request con 0x7f06f4141940 auth_method 0 2023-10-30T12:39:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f8bbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f8bbc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f8bbc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f37fe640 10 monclient(hunting): get_auth_request con 0x7f06f413e730 auth_method 0 2023-10-30T12:39:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f3fff640 10 monclient(hunting): get_auth_request con 0x7f06f40b7c10 auth_method 0 2023-10-30T12:39:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f8bbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f8bbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f8bbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.213+0000 7f06f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.216+0000 7f06f8bbc640 10 monclient(hunting): handle_auth_done global_id 17726 payload 824 2023-10-30T12:39:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.216+0000 7f06f8bbc640 1 -- 172.21.15.148:0/4087602167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06f40b7c10 msgr2=0x7f06f41da550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.216+0000 7f06f8bbc640 1 --2- 172.21.15.148:0/4087602167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06f40b7c10 0x7f06f41da550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.216+0000 7f06f8bbc640 1 -- 172.21.15.148:0/4087602167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06f413e730 msgr2=0x7f06f40b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.216+0000 7f06f8bbc640 1 --2- 172.21.15.148:0/4087602167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06f413e730 0x7f06f40b0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.216+0000 7f06f8bbc640 10 monclient: _finish_hunting 0 2023-10-30T12:39:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.216+0000 7f06f8bbc640 1 monclient: found mon.noname-c 2023-10-30T12:39:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.216+0000 7f06f8bbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.216+0000 7f06f8bbc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:39:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.216+0000 7f06f8bbc640 1 -- 172.21.15.148:0/4087602167 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f413a700 con 0x7f06f4141940 2023-10-30T12:39:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.219+0000 7f06f8bbc640 1 --2- 172.21.15.148:0/4087602167 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f411eed0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00efc0 tx=0x7f06ec009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.219+0000 7f06f0ff9640 10 client.?.objecter ms_handle_connect 0x7f06f4141940 2023-10-30T12:39:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.219+0000 7f06f0ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06f0ff9640 1 -- 172.21.15.148:0/4087602167 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec00f040 con 0x7f06f4141940 2023-10-30T12:39:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06f0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06f0ff9640 10 monclient: dump: 2023-10-30T12:39:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:50.251 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06f0ff9640 10 monclient: _finish_auth 0 2023-10-30T12:39:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06f0ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:39:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 5 monclient: authenticate success, global_id 17726 2023-10-30T12:39:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 10 monclient: _renew_subs 2023-10-30T12:39:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:50.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f413a960 con 0x7f06f4141940 2023-10-30T12:39:50.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 10 client.17726.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:50.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 10 monclient: _renew_subs 2023-10-30T12:39:50.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:50.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f411f5f0 con 0x7f06f4141940 2023-10-30T12:39:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 1 librados: init done 2023-10-30T12:39:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 10 client.17726.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 15 client.17726.objecter _session_command_op_assign -1 1 2023-10-30T12:39:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 20 client.17726.objecter _get_session osd=-1 returning homeless 2023-10-30T12:39:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 10 client.17726.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.221+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f06f4140a80 con 0x7f06f4141940 2023-10-30T12:39:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.222+0000 7f06f0ff9640 1 -- 172.21.15.148:0/4087602167 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec021070 con 0x7f06f4141940 2023-10-30T12:39:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.222+0000 7f06f0ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.222+0000 7f06f0ff9640 1 -- 172.21.15.148:0/4087602167 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4140a80 con 0x7f06f4141940 2023-10-30T12:39:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.222+0000 7f06f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.222+0000 7f06f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.222+0000 7f06f27fc640 4 set_mon_vals no callback set 2023-10-30T12:39:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.222+0000 7f06f0ff9640 10 monclient: dump: 2023-10-30T12:39:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.257 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.257 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:50.257 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.257 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:50.257 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:50.258 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:50.258 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.242+0000 7f06f0ff9640 1 -- 172.21.15.148:0/4087602167 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f06ec003990 con 0x7f06f4141940 2023-10-30T12:39:50.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.243+0000 7f06f0ff9640 1 --2- 172.21.15.148:0/4087602167 >> v2:172.21.15.148:6800/3205090179 conn(0x7f06d406c7c0 0x7f06d406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.243+0000 7f06f3fff640 1 --2- 172.21.15.148:0/4087602167 >> v2:172.21.15.148:6800/3205090179 conn(0x7f06d406c7c0 0x7f06d406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.243+0000 7f06f3fff640 10 monclient: get_auth_request con 0x7f06d406c7c0 auth_method 0 2023-10-30T12:39:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.243+0000 7f06f0ff9640 10 client.17726.objecter ms_handle_connect 0x7f06d406c7c0 2023-10-30T12:39:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.243+0000 7f06f3fff640 1 --2- 172.21.15.148:0/4087602167 >> v2:172.21.15.148:6800/3205090179 conn(0x7f06d406c7c0 0x7f06d406ec80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f06e40017f0 tx=0x7f06e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 1 -- 172.21.15.148:0/4087602167 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f06ec058df0 con 0x7f06f4141940 2023-10-30T12:39:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 10 client.17726.objecter ms_dispatch 0x7f06f4000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 3 client.17726.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 3 client.17726.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 10 client.17726.objecter checking command 1 2023-10-30T12:39:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 1 --2- 172.21.15.148:0/4087602167 >> v2:172.21.15.148:6806/2233349848 conn(0x7f06d4073ae0 0x7f06d4075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 20 client.17726.objecter _get_session s=0x7f06f4140a80 osd=3 3 2023-10-30T12:39:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 20 client.17726.objecter put_session s=0x7f06f4140a80 osd=3 3 2023-10-30T12:39:50.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 15 client.17726.objecter _session_command_op_remove -1 1 2023-10-30T12:39:50.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 20 client.17726.objecter _get_session s=0x7f06f4140a80 osd=3 3 2023-10-30T12:39:50.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 20 client.17726.objecter get_session s=0x7f06f4140a80 osd=3 3 2023-10-30T12:39:50.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 15 client.17726.objecter _session_command_op_assign 3 1 2023-10-30T12:39:50.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 20 client.17726.objecter put_session s=0x7f06f4140a80 osd=3 4 2023-10-30T12:39:50.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 10 client.17726.objecter _send_command 1 2023-10-30T12:39:50.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 1 -- 172.21.15.148:0/4087602167 --> v2:172.21.15.148:6806/2233349848 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f06ec00f040 con 0x7f06d4073ae0 2023-10-30T12:39:50.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 20 client.17726.objecter dump_active .. 0 homeless 2023-10-30T12:39:50.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 1 -- 172.21.15.148:0/4087602167 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06ec009d70 con 0x7f06f4141940 2023-10-30T12:39:50.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.246+0000 7f06f0ff9640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:39:50.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.247+0000 7f06f37fe640 1 --2- 172.21.15.148:0/4087602167 >> v2:172.21.15.148:6806/2233349848 conn(0x7f06d4073ae0 0x7f06d4075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.247+0000 7f06f37fe640 10 monclient: get_auth_request con 0x7f06d4073ae0 auth_method 0 2023-10-30T12:39:50.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.250+0000 7f06f0ff9640 10 client.17726.objecter ms_handle_connect 0x7f06d4073ae0 2023-10-30T12:39:50.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.250+0000 7f06f37fe640 1 --2- 172.21.15.148:0/4087602167 >> v2:172.21.15.148:6806/2233349848 conn(0x7f06d4073ae0 0x7f06d4075f80 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:50.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.272+0000 7f06f0ff9640 1 -- 172.21.15.148:0/4087602167 <== osd.3 v2:172.21.15.148:6806/2233349848 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f06ec00f040 con 0x7f06d4073ae0 2023-10-30T12:39:50.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.272+0000 7f06f0ff9640 10 client.17726.objecter ms_dispatch 0x7f06f4000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:39:50.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.272+0000 7f06f0ff9640 10 client.17726.objecter _finish_command 1 = system:0 2023-10-30T12:39:50.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.272+0000 7f06f0ff9640 20 client.17726.objecter put_session s=0x7f06f4140a80 osd=3 4 2023-10-30T12:39:50.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.272+0000 7f06f0ff9640 15 client.17726.objecter _session_command_op_remove 3 1 2023-10-30T12:39:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 10 client.17726.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:39:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 15 client.17726.objecter _session_command_op_assign -1 2 2023-10-30T12:39:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 20 client.17726.objecter _get_session s=0x7f06f4140a80 osd=3 3 2023-10-30T12:39:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 20 client.17726.objecter put_session s=0x7f06f4140a80 osd=3 3 2023-10-30T12:39:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 20 client.17726.objecter _get_session s=0x7f06f4140a80 osd=3 3 2023-10-30T12:39:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 15 client.17726.objecter _session_command_op_remove -1 2 2023-10-30T12:39:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 20 client.17726.objecter get_session s=0x7f06f4140a80 osd=3 3 2023-10-30T12:39:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 15 client.17726.objecter _session_command_op_assign 3 2 2023-10-30T12:39:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 20 client.17726.objecter put_session s=0x7f06f4140a80 osd=3 4 2023-10-30T12:39:50.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 10 client.17726.objecter _send_command 2 2023-10-30T12:39:50.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.284+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 --> v2:172.21.15.148:6806/2233349848 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f06f40a97c0 con 0x7f06d4073ae0 2023-10-30T12:39:50.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06f0ff9640 1 -- 172.21.15.148:0/4087602167 <== osd.3 v2:172.21.15.148:6806/2233349848 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f06f40a97c0 con 0x7f06d4073ae0 2023-10-30T12:39:50.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06f0ff9640 10 client.17726.objecter ms_dispatch 0x7f06f4000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:39:50.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06f0ff9640 10 client.17726.objecter _finish_command 2 = system:0 2023-10-30T12:39:50.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06f0ff9640 20 client.17726.objecter put_session s=0x7f06f4140a80 osd=3 4 2023-10-30T12:39:50.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06f0ff9640 15 client.17726.objecter _session_command_op_remove 3 2 2023-10-30T12:39:50.302 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T12:39:50.302 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:39:50.302 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T12:39:50.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 10 librados: watch_flush enter 2023-10-30T12:39:50.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 10 librados: watch_flush exit 2023-10-30T12:39:50.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 10 client.17726.objecter close_session for osd.3 2023-10-30T12:39:50.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 >> v2:172.21.15.148:6806/2233349848 conn(0x7f06d4073ae0 msgr2=0x7f06d4075f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 1 --2- 172.21.15.148:0/4087602167 >> v2:172.21.15.148:6806/2233349848 conn(0x7f06d4073ae0 0x7f06d4075f80 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 20 client.17726.objecter put_session s=0x7f06f4140a80 osd=3 1 2023-10-30T12:39:50.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 20 client.17726.objecter shutdown clearing up homeless session... 2023-10-30T12:39:50.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 10 client.17726.objecter successfully canceled tick 2023-10-30T12:39:50.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 >> v2:172.21.15.148:6800/3205090179 conn(0x7f06d406c7c0 msgr2=0x7f06d406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 1 --2- 172.21.15.148:0/4087602167 >> v2:172.21.15.148:6800/3205090179 conn(0x7f06d406c7c0 0x7f06d406ec80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f06e40017f0 tx=0x7f06e4009040 comp rx=0 tx=0).stop 2023-10-30T12:39:50.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.299+0000 7f06fa61f640 10 monclient: shutdown 2023-10-30T12:39:50.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.300+0000 7f06fa61f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:50.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.300+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 msgr2=0x7f06f411eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.300+0000 7f06fa61f640 1 --2- 172.21.15.148:0/4087602167 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f06f4141940 0x7f06f411eed0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00efc0 tx=0x7f06ec009f90 comp rx=0 tx=0).stop 2023-10-30T12:39:50.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.300+0000 7f06f8bbc640 1 -- 172.21.15.148:0/4087602167 reap_dead start 2023-10-30T12:39:50.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.300+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 shutdown_connections 2023-10-30T12:39:50.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.300+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 >> 172.21.15.148:0/4087602167 conn(0x7f06f406c660 msgr2=0x7f06f413c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:50.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.300+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 shutdown_connections 2023-10-30T12:39:50.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.300+0000 7f06fa61f640 1 -- 172.21.15.148:0/4087602167 wait complete. 2023-10-30T12:39:50.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.300+0000 7f06fa61f640 1 librados: shutdown 2023-10-30T12:39:50.313 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T12:39:50.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.417+0000 7f6b5f44f640 10 monclient: get_monmap_and_config 2023-10-30T12:39:50.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.417+0000 7f6b5f44f640 10 monclient: build_initial_monmap 2023-10-30T12:39:50.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.417+0000 7f6b5f44f640 10 monclient: monmap: 2023-10-30T12:39:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:50.418816+0000 2023-10-30T12:39:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:50.418816+0000 2023-10-30T12:39:50.421 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:50.421 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.421 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:50.421 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:50.421 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:50.421 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:50.422 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:50.422 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.418+0000 7f6b5f44f640 1 Processor -- start 2023-10-30T12:39:50.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.418+0000 7f6b5f44f640 1 -- start start 2023-10-30T12:39:50.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.418+0000 7f6b5f44f640 10 monclient: init 2023-10-30T12:39:50.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:50.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-10-30T12:39:50.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6b580b7c10 0x7f6b580a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 10 monclient(hunting): picked mon.noname-c con 0x7f6b580b7c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:50.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6b58141940 0x7f6b58141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 10 monclient(hunting): picked mon.noname-a con 0x7f6b58141940 addr v2:172.21.15.148:3300/0 2023-10-30T12:39:50.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b580b1c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:50.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b580a9350 con 0x7f6b580b1c10 2023-10-30T12:39:50.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:50.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b580670a0 con 0x7f6b58141940 2023-10-30T12:39:50.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:50.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b580b7ff0 con 0x7f6b580b7c10 2023-10-30T12:39:50.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:50.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5f44f640 10 monclient(hunting): authenticate will time out at 2882.624184s 2023-10-30T12:39:50.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.419+0000 7f6b5d1eb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6b580b7c10 0x7f6b580a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.420+0000 7f6b5d1eb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6b580b7c10 0x7f6b580a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:55630/0 (socket says 172.21.15.148:55630) 2023-10-30T12:39:50.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.420+0000 7f6b5d1eb640 1 -- 172.21.15.148:0/3282161278 learned_addr learned my addr 172.21.15.148:0/3282161278 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:50.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.420+0000 7f6b5d1eb640 10 monclient(hunting): get_auth_request con 0x7f6b580b7c10 auth_method 0 2023-10-30T12:39:50.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.420+0000 7f6b5d1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:50.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.420+0000 7f6b5d1eb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:50.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.420+0000 7f6b5d1eb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:50.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.421+0000 7f6b5d1eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:50.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.421+0000 7f6b5d1eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:50.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.421+0000 7f6b5d1eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:50.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.423+0000 7f6b5d1eb640 10 monclient(hunting): handle_auth_done global_id 22099 payload 274 2023-10-30T12:39:50.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.423+0000 7f6b5d1eb640 1 -- 172.21.15.148:0/3282161278 >> v2:172.21.15.148:3300/0 conn(0x7f6b58141940 msgr2=0x7f6b58141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:50.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.423+0000 7f6b5d1eb640 1 --2- 172.21.15.148:0/3282161278 >> v2:172.21.15.148:3300/0 conn(0x7f6b58141940 0x7f6b58141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.423+0000 7f6b5d1eb640 1 -- 172.21.15.148:0/3282161278 >> v1:172.21.15.150:6789/0 conn(0x7f6b580b1c10 legacy=0x7f6b580b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.423+0000 7f6b5d1eb640 10 monclient: _finish_hunting 0 2023-10-30T12:39:50.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.423+0000 7f6b5d1eb640 1 monclient: found mon.noname-c 2023-10-30T12:39:50.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.423+0000 7f6b5d1eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:50.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.423+0000 7f6b5d1eb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:39:50.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.423+0000 7f6b5d1eb640 1 -- 172.21.15.148:0/3282161278 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580aa6a0 con 0x7f6b580b7c10 2023-10-30T12:39:50.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.428+0000 7f6b5d1eb640 1 --2- 172.21.15.148:0/3282161278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b7c10 0x7f6b580a8aa0 secure :-1 s=READY pgs=5444 cs=0 l=1 rev1=1 crypto rx=0x7f6b580b4d80 tx=0x7f6b5402b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1c190bfcf0888f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:50.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.433+0000 7f6b4ffff640 1 -- 172.21.15.148:0/3282161278 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54031880 con 0x7f6b580b7c10 2023-10-30T12:39:50.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.433+0000 7f6b4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:50.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.433+0000 7f6b4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:50.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.433+0000 7f6b4ffff640 10 monclient: dump: 2023-10-30T12:39:50.437 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:50.437 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.437 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.437 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.438 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:50.438 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.438 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:50.438 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:50.439 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:50.439 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.433+0000 7f6b4ffff640 10 monclient: _finish_auth 0 2023-10-30T12:39:50.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.433+0000 7f6b4ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:39:50.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.433+0000 7f6b4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:50.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.433+0000 7f6b5f44f640 5 monclient: authenticate success, global_id 22099 2023-10-30T12:39:50.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.433+0000 7f6b4ffff640 1 -- 172.21.15.148:0/3282161278 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b540319f0 con 0x7f6b580b7c10 2023-10-30T12:39:50.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.433+0000 7f6b4ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:50.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:50.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b4ffff640 1 -- 172.21.15.148:0/3282161278 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54031d90 con 0x7f6b580b7c10 2023-10-30T12:39:50.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 4 set_mon_vals no callback set 2023-10-30T12:39:50.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:50.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:50.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b4ffff640 10 monclient: dump: 2023-10-30T12:39:50.441 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:50.441 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.441 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.442 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.442 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:50.442 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.442 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:50.442 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:50.443 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:50.443 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 10 monclient: shutdown 2023-10-30T12:39:50.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:50.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 1 -- 172.21.15.148:0/3282161278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b7c10 msgr2=0x7f6b580a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 1 --2- 172.21.15.148:0/3282161278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b7c10 0x7f6b580a8aa0 secure :-1 s=READY pgs=5444 cs=0 l=1 rev1=1 crypto rx=0x7f6b580b4d80 tx=0x7f6b5402b920 comp rx=0 tx=0).stop 2023-10-30T12:39:50.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 1 -- 172.21.15.148:0/3282161278 shutdown_connections 2023-10-30T12:39:50.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 1 --2- 172.21.15.148:0/3282161278 >> v2:172.21.15.148:3300/0 conn(0x7f6b58141940 0x7f6b58141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 1 --2- 172.21.15.148:0/3282161278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b7c10 0x7f6b580a8aa0 unknown :-1 s=CLOSED pgs=5444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 1 -- 172.21.15.148:0/3282161278 >> 172.21.15.148:0/3282161278 conn(0x7f6b5806c660 msgr2=0x7f6b580b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:50.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 1 -- 172.21.15.148:0/3282161278 shutdown_connections 2023-10-30T12:39:50.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.434+0000 7f6b5f44f640 1 -- 172.21.15.148:0/3282161278 wait complete. 2023-10-30T12:39:50.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.435+0000 7f6b5f44f640 10 monclient: build_initial_monmap 2023-10-30T12:39:50.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.435+0000 7f6b5f44f640 10 monclient: monmap: 2023-10-30T12:39:50.449 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:50.449 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:50.449 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:50.449 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:50.449 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:50.449 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:50.450 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:50.450 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:50.450 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:50.450 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.435+0000 7f6b5f44f640 1 librados: starting msgr at 2023-10-30T12:39:50.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.435+0000 7f6b5f44f640 1 librados: starting objecter 2023-10-30T12:39:50.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.435+0000 7f6b5f44f640 1 Processor -- start 2023-10-30T12:39:50.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.435+0000 7f6b5f44f640 1 -- start start 2023-10-30T12:39:50.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.435+0000 7f6b5f44f640 1 librados: setting wanted keys 2023-10-30T12:39:50.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.435+0000 7f6b5f44f640 1 librados: calling monclient init 2023-10-30T12:39:50.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.435+0000 7f6b5f44f640 10 monclient: init 2023-10-30T12:39:50.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:50.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:39:50.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b1c10 0x7f6b581da550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 10 monclient(hunting): picked mon.noname-b con 0x7f6b580b1c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:50.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f6b580b7c10 0x7f6b581dacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 10 monclient(hunting): picked mon.noname-c con 0x7f6b580b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:50.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b58141940 0x7f6b580accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 10 monclient(hunting): picked mon.noname-a con 0x7f6b58141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:50.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:50.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580a8690 con 0x7f6b58141940 2023-10-30T12:39:50.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:50.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b581db230 con 0x7f6b580b1c10 2023-10-30T12:39:50.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:50.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580b0c50 con 0x7f6b580b7c10 2023-10-30T12:39:50.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:50.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.436+0000 7f6b5f44f640 10 monclient(hunting): authenticate will time out at 2882.641408s 2023-10-30T12:39:50.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d1eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b1c10 0x7f6b581da550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d9ec640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b58141940 0x7f6b580accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5c9ea640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f6b580b7c10 0x7f6b581dacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d1eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b1c10 0x7f6b581da550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:55642/0 (socket says 172.21.15.148:55642) 2023-10-30T12:39:50.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d1eb640 1 -- 172.21.15.148:0/4111162988 learned_addr learned my addr 172.21.15.148:0/4111162988 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:50.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d1eb640 10 monclient(hunting): get_auth_request con 0x7f6b580b1c10 auth_method 0 2023-10-30T12:39:50.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:50.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d1eb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:50.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d1eb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:50.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d9ec640 10 monclient(hunting): get_auth_request con 0x7f6b58141940 auth_method 0 2023-10-30T12:39:50.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:50.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d9ec640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:50.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d9ec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:50.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5c9ea640 10 monclient(hunting): get_auth_request con 0x7f6b580b7c10 auth_method 0 2023-10-30T12:39:50.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5c9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:50.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5c9ea640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:50.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5c9ea640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:50.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d1eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d1eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.437+0000 7f6b5d1eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.438+0000 7f6b5d9ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.438+0000 7f6b5d9ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.438+0000 7f6b5d9ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.438+0000 7f6b5c9ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.438+0000 7f6b5c9ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.438+0000 7f6b5c9ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.440+0000 7f6b5d1eb640 10 monclient(hunting): handle_auth_done global_id 22102 payload 824 2023-10-30T12:39:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.440+0000 7f6b5d1eb640 1 -- 172.21.15.148:0/4111162988 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f6b580b7c10 msgr2=0x7f6b581dacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.440+0000 7f6b5d1eb640 1 --2- 172.21.15.148:0/4111162988 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f6b580b7c10 0x7f6b581dacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.440+0000 7f6b5d1eb640 1 -- 172.21.15.148:0/4111162988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b58141940 msgr2=0x7f6b580accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.440+0000 7f6b5d1eb640 1 --2- 172.21.15.148:0/4111162988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b58141940 0x7f6b580accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.440+0000 7f6b5d1eb640 10 monclient: _finish_hunting 0 2023-10-30T12:39:50.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.440+0000 7f6b5d1eb640 1 monclient: found mon.noname-b 2023-10-30T12:39:50.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.440+0000 7f6b5d1eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:50.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.440+0000 7f6b5d1eb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:39:50.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.440+0000 7f6b5d1eb640 1 -- 172.21.15.148:0/4111162988 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580ad200 con 0x7f6b580b1c10 2023-10-30T12:39:50.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.442+0000 7f6b5d1eb640 1 --2- 172.21.15.148:0/4111162988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b1c10 0x7f6b581da550 secure :-1 s=READY pgs=5445 cs=0 l=1 rev1=1 crypto rx=0x7f6b5402b960 tx=0x7f6b540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:50.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.442+0000 7f6b4dffb640 10 client.?.objecter ms_handle_connect 0x7f6b580b1c10 2023-10-30T12:39:50.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.442+0000 7f6b4dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:50.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 1 -- 172.21.15.148:0/4111162988 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54031850 con 0x7f6b580b1c10 2023-10-30T12:39:50.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:50.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:50.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 10 monclient: dump: 2023-10-30T12:39:50.462 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:50.462 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.463 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.463 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.463 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:50.463 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.463 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:50.463 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:50.464 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:50.464 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 10 monclient: _finish_auth 0 2023-10-30T12:39:50.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:39:50.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:50.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 1 -- 172.21.15.148:0/4111162988 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5403e030 con 0x7f6b580b1c10 2023-10-30T12:39:50.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b5f44f640 5 monclient: authenticate success, global_id 22102 2023-10-30T12:39:50.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b5f44f640 10 monclient: _renew_subs 2023-10-30T12:39:50.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b5f44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:50.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b5f44f640 1 -- 172.21.15.148:0/4111162988 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b580aeb90 con 0x7f6b580b1c10 2023-10-30T12:39:50.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:50.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 1 -- 172.21.15.148:0/4111162988 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54044630 con 0x7f6b580b1c10 2023-10-30T12:39:50.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:50.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:50.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4dffb640 10 monclient: dump: 2023-10-30T12:39:50.466 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:50.466 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.467 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.467 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:50.467 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:50.467 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.467 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:50.468 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:50.468 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:50.468 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b4f7fe640 4 set_mon_vals no callback set 2023-10-30T12:39:50.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b5f44f640 10 client.22102.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:50.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b5f44f640 10 monclient: _renew_subs 2023-10-30T12:39:50.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b5f44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:50.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b5f44f640 1 -- 172.21.15.148:0/4111162988 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b58139470 con 0x7f6b580b1c10 2023-10-30T12:39:50.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.446+0000 7f6b5f44f640 1 librados: init done 2023-10-30T12:39:50.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.447+0000 7f6b5f44f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:50.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.447+0000 7f6b5f44f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:50.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.447+0000 7f6b5f44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:50.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.447+0000 7f6b5f44f640 1 -- 172.21.15.148:0/4111162988 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b581daa90 con 0x7f6b580b1c10 2023-10-30T12:39:50.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.465+0000 7f6b4dffb640 1 -- 172.21.15.148:0/4111162988 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f6b540447a0 con 0x7f6b580b1c10 2023-10-30T12:39:50.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.466+0000 7f6b4dffb640 1 --2- 172.21.15.148:0/4111162988 >> v2:172.21.15.148:6800/3205090179 conn(0x7f6b3c06c7c0 0x7f6b3c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:50.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.466+0000 7f6b5c9ea640 1 --2- 172.21.15.148:0/4111162988 >> v2:172.21.15.148:6800/3205090179 conn(0x7f6b3c06c7c0 0x7f6b3c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:50.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.466+0000 7f6b5c9ea640 10 monclient: get_auth_request con 0x7f6b3c06c7c0 auth_method 0 2023-10-30T12:39:50.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.466+0000 7f6b4dffb640 10 client.22102.objecter ms_handle_connect 0x7f6b3c06c7c0 2023-10-30T12:39:50.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.466+0000 7f6b5c9ea640 1 --2- 172.21.15.148:0/4111162988 >> v2:172.21.15.148:6800/3205090179 conn(0x7f6b3c06c7c0 0x7f6b3c06ec80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6b581c83a0 tx=0x7f6b48009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.689+0000 7f6b4dffb640 1 -- 172.21.15.148:0/4111162988 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f6b5402d040 con 0x7f6b580b1c10 2023-10-30T12:39:50.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.689+0000 7f6b4dffb640 10 client.22102.objecter ms_dispatch 0x7f6b58000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:50.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.689+0000 7f6b4dffb640 3 client.22102.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:50.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.689+0000 7f6b4dffb640 3 client.22102.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:50.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.689+0000 7f6b4dffb640 20 client.22102.objecter dump_active .. 0 homeless 2023-10-30T12:39:50.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.698+0000 7f6b4dffb640 1 -- 172.21.15.148:0/4111162988 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6b5407ab80 con 0x7f6b580b1c10 2023-10-30T12:39:50.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.698+0000 7f6b4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:50.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.698+0000 7f6b4dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:50.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.873+0000 7f6b5f44f640 1 -- 172.21.15.148:0/4111162988 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b580aa6a0 con 0x7f6b3c06c7c0 2023-10-30T12:39:50.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.873+0000 7f6b4dffb640 1 -- 172.21.15.148:0/4111162988 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6b580aa6a0 con 0x7f6b3c06c7c0 2023-10-30T12:39:50.875 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T12:39:50.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 10 librados: watch_flush enter 2023-10-30T12:39:50.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 10 librados: watch_flush exit 2023-10-30T12:39:50.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 20 client.22102.objecter shutdown clearing up homeless session... 2023-10-30T12:39:50.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 10 client.22102.objecter successfully canceled tick 2023-10-30T12:39:50.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 -- 172.21.15.148:0/4111162988 >> v2:172.21.15.148:6800/3205090179 conn(0x7f6b3c06c7c0 msgr2=0x7f6b3c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 --2- 172.21.15.148:0/4111162988 >> v2:172.21.15.148:6800/3205090179 conn(0x7f6b3c06c7c0 0x7f6b3c06ec80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6b581c83a0 tx=0x7f6b48009060 comp rx=0 tx=0).stop 2023-10-30T12:39:50.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 10 monclient: shutdown 2023-10-30T12:39:50.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:50.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 -- 172.21.15.148:0/4111162988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b1c10 msgr2=0x7f6b581da550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:50.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 --2- 172.21.15.148:0/4111162988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b1c10 0x7f6b581da550 secure :-1 s=READY pgs=5445 cs=0 l=1 rev1=1 crypto rx=0x7f6b5402b960 tx=0x7f6b540046c0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 -- 172.21.15.148:0/4111162988 shutdown_connections 2023-10-30T12:39:50.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 --2- 172.21.15.148:0/4111162988 >> v2:172.21.15.148:6800/3205090179 conn(0x7f6b3c06c7c0 0x7f6b3c06ec80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 --2- 172.21.15.148:0/4111162988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b58141940 0x7f6b580accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 --2- 172.21.15.148:0/4111162988 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f6b580b7c10 0x7f6b581dacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 --2- 172.21.15.148:0/4111162988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b580b1c10 0x7f6b581da550 unknown :-1 s=CLOSED pgs=5445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:50.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 -- 172.21.15.148:0/4111162988 >> 172.21.15.148:0/4111162988 conn(0x7f6b5806c660 msgr2=0x7f6b5813ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:50.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.875+0000 7f6b5f44f640 1 -- 172.21.15.148:0/4111162988 shutdown_connections 2023-10-30T12:39:50.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.876+0000 7f6b5f44f640 1 -- 172.21.15.148:0/4111162988 wait complete. 2023-10-30T12:39:50.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.876+0000 7f6b5f44f640 1 librados: shutdown 2023-10-30T12:39:50.888 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T12:39:50.888 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:39:50.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.991+0000 7f3e52abb640 10 monclient: get_monmap_and_config 2023-10-30T12:39:50.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.991+0000 7f3e52abb640 10 monclient: build_initial_monmap 2023-10-30T12:39:50.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.992+0000 7f3e52abb640 10 monclient: monmap: 2023-10-30T12:39:50.994 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:50.994 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:50.994 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:50.993040+0000 2023-10-30T12:39:50.995 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:50.993040+0000 2023-10-30T12:39:50.995 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:50.995 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:50.995 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:50.995 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:50.996 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:50.996 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:50.996 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:50.996 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:50.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.992+0000 7f3e52abb640 1 Processor -- start 2023-10-30T12:39:50.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.992+0000 7f3e52abb640 1 -- start start 2023-10-30T12:39:50.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.992+0000 7f3e52abb640 10 monclient: init 2023-10-30T12:39:50.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:50.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-10-30T12:39:51.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3e4c0a9210 0x7f3e4c071340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4c0a9210 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:51.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e4c13b1d0 0x7f3e4c13b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 10 monclient(hunting): picked mon.noname-c con 0x7f3e4c13b1d0 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:51.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e4c0aece0 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:51.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0aa630 con 0x7f3e4c0aece0 2023-10-30T12:39:51.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c067340 con 0x7f3e4c13b1d0 2023-10-30T12:39:51.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c0aa390 con 0x7f3e4c0a9210 2023-10-30T12:39:51.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:51.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e52abb640 10 monclient(hunting): authenticate will time out at 2883.198298s 2023-10-30T12:39:51.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e51058640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3e4c0aece0 0x7f3e4c0ad7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55828/0 (socket says 172.21.15.148:55828) 2023-10-30T12:39:51.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e51058640 1 -- 172.21.15.148:0/843179825 learned_addr learned my addr 172.21.15.148:0/843179825 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:39:51.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.993+0000 7f3e50857640 1 --2- 172.21.15.148:0/843179825 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3e4c0a9210 0x7f3e4c071340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e4bfff640 1 --2- 172.21.15.148:0/843179825 >> v2:172.21.15.150:3300/0 conn(0x7f3e4c13b1d0 0x7f3e4c13b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e50857640 10 monclient(hunting): get_auth_request con 0x7f3e4c0a9210 auth_method 0 2023-10-30T12:39:51.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e50857640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e50857640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e50857640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e4bfff640 10 monclient(hunting): get_auth_request con 0x7f3e4c13b1d0 auth_method 0 2023-10-30T12:39:51.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e4bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e4bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e50857640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e50857640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e50857640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.994+0000 7f3e4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4bfff640 10 monclient(hunting): handle_auth_done global_id 22105 payload 274 2023-10-30T12:39:51.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4bfff640 1 -- 172.21.15.148:0/843179825 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3e4c0a9210 msgr2=0x7f3e4c071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:51.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4bfff640 1 --2- 172.21.15.148:0/843179825 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3e4c0a9210 0x7f3e4c071340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4bfff640 1 -- 172.21.15.148:0/843179825 >> v1:172.21.15.148:6789/0 conn(0x7f3e4c0aece0 legacy=0x7f3e4c0ad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4b7fe640 1 -- 172.21.15.148:0/843179825 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310496463 0 0) 0x7f3e34001110 con 0x7f3e4c0aece0 2023-10-30T12:39:51.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:39:51.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4bfff640 1 monclient: found mon.noname-c 2023-10-30T12:39:51.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:51.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:39:51.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4bfff640 1 -- 172.21.15.148:0/843179825 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c071de0 con 0x7f3e4c13b1d0 2023-10-30T12:39:51.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.996+0000 7f3e4b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:39:51.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:50.998+0000 7f3e4bfff640 1 --2- 172.21.15.148:0/843179825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e4c13b1d0 0x7f3e4c13b650 secure :-1 s=READY pgs=5446 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0097a0 tx=0x7f3e3c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dca776667a021ff4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 1 -- 172.21.15.148:0/843179825 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c031830 con 0x7f3e4c13b1d0 2023-10-30T12:39:51.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:51.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 10 monclient: dump: 2023-10-30T12:39:51.016 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.016 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.016 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.016 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.016 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.017 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.017 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.017 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.017 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.018 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:39:51.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:39:51.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:51.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 1 -- 172.21.15.148:0/843179825 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c02d040 con 0x7f3e4c13b1d0 2023-10-30T12:39:51.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:51.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 1 -- 172.21.15.148:0/843179825 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c031bc0 con 0x7f3e4c13b1d0 2023-10-30T12:39:51.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e52abb640 5 monclient: authenticate success, global_id 22105 2023-10-30T12:39:51.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e52abb640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:51.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e52abb640 4 set_mon_vals no callback set 2023-10-30T12:39:51.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:51.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.001+0000 7f3e4b7fe640 10 monclient: dump: 2023-10-30T12:39:51.020 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.020 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.021 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.021 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.021 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.021 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.021 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.022 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.022 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.022 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.002+0000 7f3e52abb640 10 monclient: shutdown 2023-10-30T12:39:51.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.002+0000 7f3e52abb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:51.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.002+0000 7f3e52abb640 1 -- 172.21.15.148:0/843179825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e4c13b1d0 msgr2=0x7f3e4c13b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.002+0000 7f3e52abb640 1 --2- 172.21.15.148:0/843179825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e4c13b1d0 0x7f3e4c13b650 secure :-1 s=READY pgs=5446 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0097a0 tx=0x7f3e3c02b8d0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.002+0000 7f3e52abb640 1 -- 172.21.15.148:0/843179825 shutdown_connections 2023-10-30T12:39:51.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.002+0000 7f3e52abb640 1 --2- 172.21.15.148:0/843179825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e4c13b1d0 0x7f3e4c13b650 unknown :-1 s=CLOSED pgs=5446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.002+0000 7f3e52abb640 1 --2- 172.21.15.148:0/843179825 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3e4c0a9210 0x7f3e4c071340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.002+0000 7f3e52abb640 1 -- 172.21.15.148:0/843179825 >> 172.21.15.148:0/843179825 conn(0x7f3e4c0b3260 msgr2=0x7f3e4c0b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:51.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.002+0000 7f3e52abb640 1 -- 172.21.15.148:0/843179825 shutdown_connections 2023-10-30T12:39:51.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.002+0000 7f3e52abb640 1 -- 172.21.15.148:0/843179825 wait complete. 2023-10-30T12:39:51.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.003+0000 7f3e52abb640 10 monclient: build_initial_monmap 2023-10-30T12:39:51.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.003+0000 7f3e52abb640 10 monclient: monmap: 2023-10-30T12:39:51.025 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:51.025 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:51.025 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:51.026 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:51.026 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:51.026 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:51.026 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:51.027 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:51.027 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:51.027 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.003+0000 7f3e52abb640 1 librados: starting msgr at 2023-10-30T12:39:51.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.003+0000 7f3e52abb640 1 librados: starting objecter 2023-10-30T12:39:51.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.003+0000 7f3e52abb640 1 Processor -- start 2023-10-30T12:39:51.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.003+0000 7f3e52abb640 1 -- start start 2023-10-30T12:39:51.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.003+0000 7f3e52abb640 1 librados: setting wanted keys 2023-10-30T12:39:51.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.003+0000 7f3e52abb640 1 librados: calling monclient init 2023-10-30T12:39:51.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.003+0000 7f3e52abb640 10 monclient: init 2023-10-30T12:39:51.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:51.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:39:51.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e4c0a9210 0x7f3e4c1338f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 10 monclient(hunting): picked mon.noname-a con 0x7f3e4c0a9210 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:51.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e4c0aece0 0x7f3e4c133fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4c0aece0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:51.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3e4c13b1d0 0x7f3e4c1d4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 10 monclient(hunting): picked mon.noname-c con 0x7f3e4c13b1d0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:51.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c0ae650 con 0x7f3e4c0a9210 2023-10-30T12:39:51.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c1344f0 con 0x7f3e4c0aece0 2023-10-30T12:39:51.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c135d50 con 0x7f3e4c13b1d0 2023-10-30T12:39:51.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:51.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e52abb640 10 monclient(hunting): authenticate will time out at 2883.209250s 2023-10-30T12:39:51.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e50857640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e4c0a9210 0x7f3e4c1338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.004+0000 7f3e51058640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3e4c13b1d0 0x7f3e4c1d4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e4bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e4c0aece0 0x7f3e4c133fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e51058640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3e4c13b1d0 0x7f3e4c1d4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:51530/0 (socket says 172.21.15.148:51530) 2023-10-30T12:39:51.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e51058640 1 -- 172.21.15.148:0/1629944874 learned_addr learned my addr 172.21.15.148:0/1629944874 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:51.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e51058640 10 monclient(hunting): get_auth_request con 0x7f3e4c13b1d0 auth_method 0 2023-10-30T12:39:51.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e51058640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e51058640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e51058640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e50857640 10 monclient(hunting): get_auth_request con 0x7f3e4c0a9210 auth_method 0 2023-10-30T12:39:51.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e50857640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e50857640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e50857640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e4bfff640 10 monclient(hunting): get_auth_request con 0x7f3e4c0aece0 auth_method 0 2023-10-30T12:39:51.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e4bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e4bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e50857640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e50857640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e50857640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e51058640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e51058640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.005+0000 7f3e51058640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.008+0000 7f3e4bfff640 10 monclient(hunting): handle_auth_done global_id 22108 payload 824 2023-10-30T12:39:51.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.008+0000 7f3e4bfff640 1 -- 172.21.15.148:0/1629944874 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3e4c13b1d0 msgr2=0x7f3e4c1d4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.008+0000 7f3e4bfff640 1 --2- 172.21.15.148:0/1629944874 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3e4c13b1d0 0x7f3e4c1d4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.008+0000 7f3e4bfff640 1 -- 172.21.15.148:0/1629944874 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e4c0a9210 msgr2=0x7f3e4c1338f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.008+0000 7f3e4bfff640 1 --2- 172.21.15.148:0/1629944874 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e4c0a9210 0x7f3e4c1338f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.008+0000 7f3e4bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:39:51.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.008+0000 7f3e4bfff640 1 monclient: found mon.noname-b 2023-10-30T12:39:51.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.008+0000 7f3e4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:51.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.008+0000 7f3e4bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:39:51.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.008+0000 7f3e4bfff640 1 -- 172.21.15.148:0/1629944874 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c134710 con 0x7f3e4c0aece0 2023-10-30T12:39:51.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.010+0000 7f3e4bfff640 1 --2- 172.21.15.148:0/1629944874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e4c0aece0 0x7f3e4c133fb0 secure :-1 s=READY pgs=5447 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c033000 tx=0x7f3e3c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.010+0000 7f3e497fa640 10 client.?.objecter ms_handle_connect 0x7f3e4c0aece0 2023-10-30T12:39:51.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.010+0000 7f3e497fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:51.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 1 -- 172.21.15.148:0/1629944874 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c03a630 con 0x7f3e4c0aece0 2023-10-30T12:39:51.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:51.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 10 monclient: dump: 2023-10-30T12:39:51.042 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.043 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.043 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.043 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.043 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.043 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.044 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.044 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.044 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.044 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 10 monclient: _finish_auth 0 2023-10-30T12:39:51.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 10 monclient: _check_auth_tickets 2023-10-30T12:39:51.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:51.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 1 -- 172.21.15.148:0/1629944874 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c03f050 con 0x7f3e4c0aece0 2023-10-30T12:39:51.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:51.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 1 -- 172.21.15.148:0/1629944874 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c03aa00 con 0x7f3e4c0aece0 2023-10-30T12:39:51.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e52abb640 5 monclient: authenticate success, global_id 22108 2023-10-30T12:39:51.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:51.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e497fa640 10 monclient: dump: 2023-10-30T12:39:51.046 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.047 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.047 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.047 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.047 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.047 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.047 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.048 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.048 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.048 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 10 monclient: _renew_subs 2023-10-30T12:39:51.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.013+0000 7f3e4affd640 4 set_mon_vals no callback set 2023-10-30T12:39:51.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:51.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4c070e00 con 0x7f3e4c0aece0 2023-10-30T12:39:51.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 10 client.22108.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:51.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 10 monclient: _renew_subs 2023-10-30T12:39:51.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:51.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4c1d4e90 con 0x7f3e4c0aece0 2023-10-30T12:39:51.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 1 librados: init done 2023-10-30T12:39:51.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 10 client.22108.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:51.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 15 client.22108.objecter _session_command_op_assign -1 1 2023-10-30T12:39:51.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 20 client.22108.objecter _get_session osd=-1 returning homeless 2023-10-30T12:39:51.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 10 client.22108.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:51.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:51.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.014+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3e4c0aaaa0 con 0x7f3e4c0aece0 2023-10-30T12:39:51.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.033+0000 7f3e497fa640 1 -- 172.21.15.148:0/1629944874 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f3e3c03ab70 con 0x7f3e4c0aece0 2023-10-30T12:39:51.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.033+0000 7f3e497fa640 1 --2- 172.21.15.148:0/1629944874 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3e2c06c810 0x7f3e2c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.033+0000 7f3e50857640 1 --2- 172.21.15.148:0/1629944874 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3e2c06c810 0x7f3e2c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.033+0000 7f3e50857640 10 monclient: get_auth_request con 0x7f3e2c06c810 auth_method 0 2023-10-30T12:39:51.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.034+0000 7f3e497fa640 10 client.22108.objecter ms_handle_connect 0x7f3e2c06c810 2023-10-30T12:39:51.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.034+0000 7f3e50857640 1 --2- 172.21.15.148:0/1629944874 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3e2c06c810 0x7f3e2c06ecd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3e44011340 tx=0x7f3e4400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 1 -- 172.21.15.148:0/1629944874 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f3e3c03ae20 con 0x7f3e4c0aece0 2023-10-30T12:39:51.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 10 client.22108.objecter ms_dispatch 0x7f3e4c000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:51.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 3 client.22108.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:51.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 3 client.22108.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:51.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 10 client.22108.objecter checking command 1 2023-10-30T12:39:51.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 1 --2- 172.21.15.148:0/1629944874 >> v2:172.21.15.150:6808/2836839739 conn(0x7f3e2c073b30 0x7f3e2c075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 20 client.22108.objecter _get_session s=0x7f3e3c03aa00 osd=4 3 2023-10-30T12:39:51.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 20 client.22108.objecter put_session s=0x7f3e3c03aa00 osd=4 3 2023-10-30T12:39:51.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 15 client.22108.objecter _session_command_op_remove -1 1 2023-10-30T12:39:51.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 20 client.22108.objecter _get_session s=0x7f3e3c03aa00 osd=4 3 2023-10-30T12:39:51.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 20 client.22108.objecter get_session s=0x7f3e3c03aa00 osd=4 3 2023-10-30T12:39:51.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 15 client.22108.objecter _session_command_op_assign 4 1 2023-10-30T12:39:51.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 20 client.22108.objecter put_session s=0x7f3e3c03aa00 osd=4 4 2023-10-30T12:39:51.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 10 client.22108.objecter _send_command 1 2023-10-30T12:39:51.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 1 -- 172.21.15.148:0/1629944874 --> v2:172.21.15.150:6808/2836839739 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e3c03a630 con 0x7f3e2c073b30 2023-10-30T12:39:51.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 20 client.22108.objecter dump_active .. 0 homeless 2023-10-30T12:39:51.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 1 -- 172.21.15.148:0/1629944874 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e3c009400 con 0x7f3e4c0aece0 2023-10-30T12:39:51.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.035+0000 7f3e497fa640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:39:51.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.036+0000 7f3e51058640 1 --2- 172.21.15.148:0/1629944874 >> v2:172.21.15.150:6808/2836839739 conn(0x7f3e2c073b30 0x7f3e2c075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.036+0000 7f3e51058640 10 monclient: get_auth_request con 0x7f3e2c073b30 auth_method 0 2023-10-30T12:39:51.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.039+0000 7f3e51058640 1 --2- 172.21.15.148:0/1629944874 >> v2:172.21.15.150:6808/2836839739 conn(0x7f3e2c073b30 0x7f3e2c075fd0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.039+0000 7f3e497fa640 10 client.22108.objecter ms_handle_connect 0x7f3e2c073b30 2023-10-30T12:39:51.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.062+0000 7f3e497fa640 1 -- 172.21.15.148:0/1629944874 <== osd.4 v2:172.21.15.150:6808/2836839739 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f3e3c03a630 con 0x7f3e2c073b30 2023-10-30T12:39:51.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.062+0000 7f3e497fa640 10 client.22108.objecter ms_dispatch 0x7f3e4c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:39:51.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.062+0000 7f3e497fa640 10 client.22108.objecter _finish_command 1 = system:0 2023-10-30T12:39:51.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.062+0000 7f3e497fa640 20 client.22108.objecter put_session s=0x7f3e3c03aa00 osd=4 4 2023-10-30T12:39:51.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.062+0000 7f3e497fa640 15 client.22108.objecter _session_command_op_remove 4 1 2023-10-30T12:39:51.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 10 client.22108.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:39:51.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 15 client.22108.objecter _session_command_op_assign -1 2 2023-10-30T12:39:51.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 20 client.22108.objecter _get_session s=0x7f3e3c03aa00 osd=4 3 2023-10-30T12:39:51.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 20 client.22108.objecter put_session s=0x7f3e3c03aa00 osd=4 3 2023-10-30T12:39:51.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 20 client.22108.objecter _get_session s=0x7f3e3c03aa00 osd=4 3 2023-10-30T12:39:51.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 15 client.22108.objecter _session_command_op_remove -1 2 2023-10-30T12:39:51.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 20 client.22108.objecter get_session s=0x7f3e3c03aa00 osd=4 3 2023-10-30T12:39:51.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 15 client.22108.objecter _session_command_op_assign 4 2 2023-10-30T12:39:51.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 20 client.22108.objecter put_session s=0x7f3e3c03aa00 osd=4 4 2023-10-30T12:39:51.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 10 client.22108.objecter _send_command 2 2023-10-30T12:39:51.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.074+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 --> v2:172.21.15.150:6808/2836839739 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3e4c0a95f0 con 0x7f3e2c073b30 2023-10-30T12:39:51.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e497fa640 1 -- 172.21.15.148:0/1629944874 <== osd.4 v2:172.21.15.150:6808/2836839739 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f3e34007660 con 0x7f3e2c073b30 2023-10-30T12:39:51.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e497fa640 10 client.22108.objecter ms_dispatch 0x7f3e4c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:39:51.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e497fa640 10 client.22108.objecter _finish_command 2 = system:0 2023-10-30T12:39:51.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e497fa640 20 client.22108.objecter put_session s=0x7f3e3c03aa00 osd=4 4 2023-10-30T12:39:51.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e497fa640 15 client.22108.objecter _session_command_op_remove 4 2 2023-10-30T12:39:51.090 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T12:39:51.090 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:39:51.091 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T12:39:51.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 10 librados: watch_flush enter 2023-10-30T12:39:51.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 10 librados: watch_flush exit 2023-10-30T12:39:51.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 10 client.22108.objecter close_session for osd.4 2023-10-30T12:39:51.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 >> v2:172.21.15.150:6808/2836839739 conn(0x7f3e2c073b30 msgr2=0x7f3e2c075fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 --2- 172.21.15.148:0/1629944874 >> v2:172.21.15.150:6808/2836839739 conn(0x7f3e2c073b30 0x7f3e2c075fd0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 20 client.22108.objecter put_session s=0x7f3e3c03aa00 osd=4 1 2023-10-30T12:39:51.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 20 client.22108.objecter shutdown clearing up homeless session... 2023-10-30T12:39:51.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 10 client.22108.objecter successfully canceled tick 2023-10-30T12:39:51.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3e2c06c810 msgr2=0x7f3e2c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 --2- 172.21.15.148:0/1629944874 >> v2:172.21.15.148:6800/3205090179 conn(0x7f3e2c06c810 0x7f3e2c06ecd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3e44011340 tx=0x7f3e4400b040 comp rx=0 tx=0).stop 2023-10-30T12:39:51.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 10 monclient: shutdown 2023-10-30T12:39:51.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:51.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e4c0aece0 msgr2=0x7f3e4c133fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 --2- 172.21.15.148:0/1629944874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e4c0aece0 0x7f3e4c133fb0 secure :-1 s=READY pgs=5447 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c033000 tx=0x7f3e3c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e51058640 1 -- 172.21.15.148:0/1629944874 reap_dead start 2023-10-30T12:39:51.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 shutdown_connections 2023-10-30T12:39:51.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 >> 172.21.15.148:0/1629944874 conn(0x7f3e4c0b3260 msgr2=0x7f3e4c139800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:51.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 shutdown_connections 2023-10-30T12:39:51.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 -- 172.21.15.148:0/1629944874 wait complete. 2023-10-30T12:39:51.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.088+0000 7f3e52abb640 1 librados: shutdown 2023-10-30T12:39:51.100 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T12:39:51.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.200+0000 7f76fc5b0640 10 monclient: get_monmap_and_config 2023-10-30T12:39:51.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.200+0000 7f76fc5b0640 10 monclient: build_initial_monmap 2023-10-30T12:39:51.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.200+0000 7f76fc5b0640 10 monclient: monmap: 2023-10-30T12:39:51.203 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:51.203 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.203 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:51.201490+0000 2023-10-30T12:39:51.203 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:51.201490+0000 2023-10-30T12:39:51.203 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:51.204 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.204 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:51.204 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:51.204 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:51.205 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:51.205 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:51.205 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 1 Processor -- start 2023-10-30T12:39:51.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 1 -- start start 2023-10-30T12:39:51.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 10 monclient: init 2023-10-30T12:39:51.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:51.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-10-30T12:39:51.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76f40b7c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:51.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f76f4141940 0x7f76f4141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 10 monclient(hunting): picked mon.noname-c con 0x7f76f4141940 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:51.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f413e730 0x7f76f413eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 10 monclient(hunting): picked mon.noname-b con 0x7f76f413e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:51.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f40a9350 con 0x7f76f40b7c10 2023-10-30T12:39:51.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f40670a0 con 0x7f76f4141940 2023-10-30T12:39:51.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f40b7ff0 con 0x7f76f413e730 2023-10-30T12:39:51.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:51.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.201+0000 7f76fc5b0640 10 monclient(hunting): authenticate will time out at 2883.406834s 2023-10-30T12:39:51.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76fab4d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f413e730 0x7f76f413eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76f9b4b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f76f4141940 0x7f76f4141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76fab4d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f413e730 0x7f76f413eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:51542/0 (socket says 172.21.15.148:51542) 2023-10-30T12:39:51.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76fab4d640 1 -- 172.21.15.148:0/2198532124 learned_addr learned my addr 172.21.15.148:0/2198532124 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:51.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76fab4d640 10 monclient(hunting): get_auth_request con 0x7f76f413e730 auth_method 0 2023-10-30T12:39:51.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76fab4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76fab4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76fab4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76f9b4b640 10 monclient(hunting): get_auth_request con 0x7f76f4141940 auth_method 0 2023-10-30T12:39:51.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76f9b4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76f9b4b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.202+0000 7f76f9b4b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.203+0000 7f76fab4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.203+0000 7f76fab4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.203+0000 7f76fab4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.204+0000 7f76f9b4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.204+0000 7f76f9b4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.204+0000 7f76f9b4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.205+0000 7f76fab4d640 10 monclient(hunting): handle_auth_done global_id 17738 payload 274 2023-10-30T12:39:51.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.205+0000 7f76fab4d640 1 -- 172.21.15.148:0/2198532124 >> v2:172.21.15.150:3300/0 conn(0x7f76f4141940 msgr2=0x7f76f4141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:51.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.205+0000 7f76fab4d640 1 --2- 172.21.15.148:0/2198532124 >> v2:172.21.15.150:3300/0 conn(0x7f76f4141940 0x7f76f4141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.205+0000 7f76fab4d640 1 -- 172.21.15.148:0/2198532124 >> v1:172.21.15.150:6789/0 conn(0x7f76f40b7c10 legacy=0x7f76f40a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.205+0000 7f76fab4d640 10 monclient: _finish_hunting 0 2023-10-30T12:39:51.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.205+0000 7f76fab4d640 1 monclient: found mon.noname-b 2023-10-30T12:39:51.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.205+0000 7f76fab4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:51.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.205+0000 7f76fab4d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:51.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.205+0000 7f76fab4d640 1 -- 172.21.15.148:0/2198532124 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f40aa6a0 con 0x7f76f413e730 2023-10-30T12:39:51.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.207+0000 7f76fab4d640 1 --2- 172.21.15.148:0/2198532124 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f413e730 0x7f76f413eb80 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009e50 tx=0x7f76ec02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbe5dc9830573fa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 1 -- 172.21.15.148:0/2198532124 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec02ca70 con 0x7f76f413e730 2023-10-30T12:39:51.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:51.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 10 monclient: dump: 2023-10-30T12:39:51.220 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.220 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.220 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.221 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.221 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.221 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.221 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.221 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.221 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.222 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 10 monclient: _finish_auth 0 2023-10-30T12:39:51.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 10 monclient: _check_auth_tickets 2023-10-30T12:39:51.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:51.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 1 -- 172.21.15.148:0/2198532124 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ec02cbe0 con 0x7f76f413e730 2023-10-30T12:39:51.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:51.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 1 -- 172.21.15.148:0/2198532124 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec0343f0 con 0x7f76f413e730 2023-10-30T12:39:51.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 5 monclient: authenticate success, global_id 17738 2023-10-30T12:39:51.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:51.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 4 set_mon_vals no callback set 2023-10-30T12:39:51.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:51.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76f934a640 10 monclient: dump: 2023-10-30T12:39:51.224 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.224 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.225 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.225 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.225 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.225 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.225 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.226 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.226 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.226 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 10 monclient: shutdown 2023-10-30T12:39:51.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:51.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 1 -- 172.21.15.148:0/2198532124 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f413e730 msgr2=0x7f76f413eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 1 --2- 172.21.15.148:0/2198532124 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f413e730 0x7f76f413eb80 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009e50 tx=0x7f76ec02bed0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 1 -- 172.21.15.148:0/2198532124 shutdown_connections 2023-10-30T12:39:51.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 1 --2- 172.21.15.148:0/2198532124 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f413e730 0x7f76f413eb80 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 1 --2- 172.21.15.148:0/2198532124 >> v2:172.21.15.150:3300/0 conn(0x7f76f4141940 0x7f76f4141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.211+0000 7f76fc5b0640 1 -- 172.21.15.148:0/2198532124 >> 172.21.15.148:0/2198532124 conn(0x7f76f406c660 msgr2=0x7f76f40b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:51.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.212+0000 7f76fc5b0640 1 -- 172.21.15.148:0/2198532124 shutdown_connections 2023-10-30T12:39:51.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.212+0000 7f76fc5b0640 1 -- 172.21.15.148:0/2198532124 wait complete. 2023-10-30T12:39:51.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.212+0000 7f76fc5b0640 10 monclient: build_initial_monmap 2023-10-30T12:39:51.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.212+0000 7f76fc5b0640 10 monclient: monmap: 2023-10-30T12:39:51.229 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:51.229 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:51.229 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:51.229 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:51.229 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:51.229 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:51.230 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:51.230 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:51.230 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:51.230 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 librados: starting msgr at 2023-10-30T12:39:51.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 librados: starting objecter 2023-10-30T12:39:51.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 Processor -- start 2023-10-30T12:39:51.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 -- start start 2023-10-30T12:39:51.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 librados: setting wanted keys 2023-10-30T12:39:51.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 librados: calling monclient init 2023-10-30T12:39:51.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient: init 2023-10-30T12:39:51.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:51.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:39:51.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f40b7c10 0x7f76f411f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient(hunting): picked mon.noname-c con 0x7f76f40b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:51.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76f413e730 0x7f76f41d6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient(hunting): picked mon.noname-a con 0x7f76f413e730 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:51.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f4141940 0x7f76f41d7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient(hunting): picked mon.noname-b con 0x7f76f4141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:51.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f413ec30 con 0x7f76f413e730 2023-10-30T12:39:51.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f41d69a0 con 0x7f76f4141940 2023-10-30T12:39:51.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f40afd40 con 0x7f76f40b7c10 2023-10-30T12:39:51.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:51.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.213+0000 7f76fc5b0640 10 monclient(hunting): authenticate will time out at 2883.418726s 2023-10-30T12:39:51.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fa34c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f40b7c10 0x7f76f411f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fab4d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f4141940 0x7f76f41d7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76f9b4b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76f413e730 0x7f76f41d6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fa34c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f40b7c10 0x7f76f411f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:51552/0 (socket says 172.21.15.148:51552) 2023-10-30T12:39:51.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fa34c640 1 -- 172.21.15.148:0/810715477 learned_addr learned my addr 172.21.15.148:0/810715477 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:51.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fa34c640 10 monclient(hunting): get_auth_request con 0x7f76f40b7c10 auth_method 0 2023-10-30T12:39:51.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fa34c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fa34c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fa34c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76f9b4b640 10 monclient(hunting): get_auth_request con 0x7f76f413e730 auth_method 0 2023-10-30T12:39:51.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76f9b4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76f9b4b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76f9b4b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fab4d640 10 monclient(hunting): get_auth_request con 0x7f76f4141940 auth_method 0 2023-10-30T12:39:51.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fab4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fab4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.214+0000 7f76fab4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.215+0000 7f76fa34c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.215+0000 7f76fa34c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.215+0000 7f76fa34c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.215+0000 7f76f9b4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.215+0000 7f76f9b4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.215+0000 7f76f9b4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.215+0000 7f76fab4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.215+0000 7f76fab4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.215+0000 7f76fab4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.217+0000 7f76fa34c640 10 monclient(hunting): handle_auth_done global_id 17741 payload 824 2023-10-30T12:39:51.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.217+0000 7f76fa34c640 1 -- 172.21.15.148:0/810715477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f4141940 msgr2=0x7f76f41d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.217+0000 7f76fa34c640 1 --2- 172.21.15.148:0/810715477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f4141940 0x7f76f41d7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.217+0000 7f76fa34c640 1 -- 172.21.15.148:0/810715477 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76f413e730 msgr2=0x7f76f41d6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.217+0000 7f76fa34c640 1 --2- 172.21.15.148:0/810715477 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76f413e730 0x7f76f41d6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.217+0000 7f76fa34c640 10 monclient: _finish_hunting 0 2023-10-30T12:39:51.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.217+0000 7f76fa34c640 1 monclient: found mon.noname-c 2023-10-30T12:39:51.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.217+0000 7f76fa34c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:51.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.217+0000 7f76fa34c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:39:51.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.217+0000 7f76fa34c640 1 -- 172.21.15.148:0/810715477 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f40add50 con 0x7f76f40b7c10 2023-10-30T12:39:51.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.219+0000 7f76fa34c640 1 --2- 172.21.15.148:0/810715477 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f40b7c10 0x7f76f411f140 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f76f000c480 tx=0x7f76f000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.220+0000 7f76eaffd640 10 client.?.objecter ms_handle_connect 0x7f76f40b7c10 2023-10-30T12:39:51.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.220+0000 7f76eaffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:51.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 1 -- 172.21.15.148:0/810715477 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f00052d0 con 0x7f76f40b7c10 2023-10-30T12:39:51.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:51.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 10 monclient: dump: 2023-10-30T12:39:51.244 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.244 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.244 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.244 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.244 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.245 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.245 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.245 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.245 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.245 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 10 monclient: _finish_auth 0 2023-10-30T12:39:51.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:39:51.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:51.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 1 -- 172.21.15.148:0/810715477 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f000e040 con 0x7f76f40b7c10 2023-10-30T12:39:51.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76fc5b0640 5 monclient: authenticate success, global_id 17741 2023-10-30T12:39:51.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76fc5b0640 10 monclient: _renew_subs 2023-10-30T12:39:51.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76fc5b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:51.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76fc5b0640 1 -- 172.21.15.148:0/810715477 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f411f680 con 0x7f76f40b7c10 2023-10-30T12:39:51.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:51.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 1 -- 172.21.15.148:0/810715477 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0012630 con 0x7f76f40b7c10 2023-10-30T12:39:51.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:51.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76eaffd640 10 monclient: dump: 2023-10-30T12:39:51.248 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.248 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.248 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.249 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.249 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.249 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.249 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.249 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.249 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.250 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76f8b49640 4 set_mon_vals no callback set 2023-10-30T12:39:51.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76fc5b0640 10 client.17741.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:51.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76fc5b0640 10 monclient: _renew_subs 2023-10-30T12:39:51.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76fc5b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:51.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76fc5b0640 1 -- 172.21.15.148:0/810715477 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f40b0a40 con 0x7f76f40b7c10 2023-10-30T12:39:51.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.223+0000 7f76fc5b0640 1 librados: init done 2023-10-30T12:39:51.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.224+0000 7f76fc5b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:51.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.224+0000 7f76fc5b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:51.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.224+0000 7f76fc5b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:51.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.224+0000 7f76fc5b0640 1 -- 172.21.15.148:0/810715477 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f411f9c0 con 0x7f76f40b7c10 2023-10-30T12:39:51.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.243+0000 7f76eaffd640 1 -- 172.21.15.148:0/810715477 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f76f00127a0 con 0x7f76f40b7c10 2023-10-30T12:39:51.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.243+0000 7f76eaffd640 1 --2- 172.21.15.148:0/810715477 >> v2:172.21.15.148:6800/3205090179 conn(0x7f76d806c810 0x7f76d806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.244+0000 7f76f9b4b640 1 --2- 172.21.15.148:0/810715477 >> v2:172.21.15.148:6800/3205090179 conn(0x7f76d806c810 0x7f76d806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.244+0000 7f76f9b4b640 10 monclient: get_auth_request con 0x7f76d806c810 auth_method 0 2023-10-30T12:39:51.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.244+0000 7f76f9b4b640 1 --2- 172.21.15.148:0/810715477 >> v2:172.21.15.148:6800/3205090179 conn(0x7f76d806c810 0x7f76d806ecd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f76f40af700 tx=0x7f76e4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.244+0000 7f76eaffd640 10 client.17741.objecter ms_handle_connect 0x7f76d806c810 2023-10-30T12:39:51.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.491+0000 7f76eaffd640 1 -- 172.21.15.148:0/810715477 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f76f0058350 con 0x7f76f40b7c10 2023-10-30T12:39:51.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.491+0000 7f76eaffd640 10 client.17741.objecter ms_dispatch 0x7f76f4000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:51.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.491+0000 7f76eaffd640 3 client.17741.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:51.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.491+0000 7f76eaffd640 3 client.17741.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:51.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.491+0000 7f76eaffd640 20 client.17741.objecter dump_active .. 0 homeless 2023-10-30T12:39:51.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.491+0000 7f76eaffd640 1 -- 172.21.15.148:0/810715477 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f76f005c3f0 con 0x7f76f40b7c10 2023-10-30T12:39:51.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.491+0000 7f76eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:51.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.491+0000 7f76eaffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:51.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.673+0000 7f76fc5b0640 1 -- 172.21.15.148:0/810715477 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f76f40a97e0 con 0x7f76d806c810 2023-10-30T12:39:51.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.673+0000 7f76eaffd640 1 -- 172.21.15.148:0/810715477 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f76e40119d0 con 0x7f76d806c810 2023-10-30T12:39:51.675 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T12:39:51.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 10 librados: watch_flush enter 2023-10-30T12:39:51.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 10 librados: watch_flush exit 2023-10-30T12:39:51.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 20 client.17741.objecter shutdown clearing up homeless session... 2023-10-30T12:39:51.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 10 client.17741.objecter successfully canceled tick 2023-10-30T12:39:51.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 -- 172.21.15.148:0/810715477 >> v2:172.21.15.148:6800/3205090179 conn(0x7f76d806c810 msgr2=0x7f76d806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 --2- 172.21.15.148:0/810715477 >> v2:172.21.15.148:6800/3205090179 conn(0x7f76d806c810 0x7f76d806ecd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f76f40af700 tx=0x7f76e4009190 comp rx=0 tx=0).stop 2023-10-30T12:39:51.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 10 monclient: shutdown 2023-10-30T12:39:51.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:51.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 -- 172.21.15.148:0/810715477 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f40b7c10 msgr2=0x7f76f411f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 --2- 172.21.15.148:0/810715477 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f40b7c10 0x7f76f411f140 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f76f000c480 tx=0x7f76f000c940 comp rx=0 tx=0).stop 2023-10-30T12:39:51.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 -- 172.21.15.148:0/810715477 shutdown_connections 2023-10-30T12:39:51.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 --2- 172.21.15.148:0/810715477 >> v2:172.21.15.148:6800/3205090179 conn(0x7f76d806c810 0x7f76d806ecd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 --2- 172.21.15.148:0/810715477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f4141940 0x7f76f41d7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 --2- 172.21.15.148:0/810715477 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76f413e730 0x7f76f41d6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 --2- 172.21.15.148:0/810715477 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f76f40b7c10 0x7f76f411f140 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 -- 172.21.15.148:0/810715477 >> 172.21.15.148:0/810715477 conn(0x7f76f406c660 msgr2=0x7f76f40a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:51.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 -- 172.21.15.148:0/810715477 shutdown_connections 2023-10-30T12:39:51.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 -- 172.21.15.148:0/810715477 wait complete. 2023-10-30T12:39:51.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.675+0000 7f76fc5b0640 1 librados: shutdown 2023-10-30T12:39:51.688 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T12:39:51.688 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:39:51.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.792+0000 7f5904e09640 10 monclient: get_monmap_and_config 2023-10-30T12:39:51.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.792+0000 7f5904e09640 10 monclient: build_initial_monmap 2023-10-30T12:39:51.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.792+0000 7f5904e09640 10 monclient: monmap: 2023-10-30T12:39:51.794 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:51.794 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.795 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:51.793288+0000 2023-10-30T12:39:51.795 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:51.793288+0000 2023-10-30T12:39:51.795 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:51.795 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.795 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:51.796 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:51.796 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:51.796 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:51.796 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:51.796 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.792+0000 7f5904e09640 1 Processor -- start 2023-10-30T12:39:51.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.792+0000 7f5904e09640 1 -- start start 2023-10-30T12:39:51.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.792+0000 7f5904e09640 10 monclient: init 2023-10-30T12:39:51.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:51.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-10-30T12:39:51.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59000b7c10 0x7f59000a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 10 monclient(hunting): picked mon.noname-b con 0x7f59000b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:51.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5900141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:51.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f590013e730 0x7f590013eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 10 monclient(hunting): picked mon.noname-c con 0x7f590013e730 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:51.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59000a9350 con 0x7f5900141940 2023-10-30T12:39:51.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59000670a0 con 0x7f590013e730 2023-10-30T12:39:51.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5900142470 con 0x7f59000b7c10 2023-10-30T12:39:51.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:51.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.793+0000 7f5904e09640 10 monclient(hunting): authenticate will time out at 2883.998465s 2023-10-30T12:39:51.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.794+0000 7f58fed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f590013e730 0x7f590013eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.794+0000 7f58fe59c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59000b7c10 0x7f59000a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fe59c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59000b7c10 0x7f59000a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:51556/0 (socket says 172.21.15.148:51556) 2023-10-30T12:39:51.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fe59c640 1 -- 172.21.15.148:0/2884312825 learned_addr learned my addr 172.21.15.148:0/2884312825 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:51.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fe59c640 10 monclient(hunting): get_auth_request con 0x7f59000b7c10 auth_method 0 2023-10-30T12:39:51.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fe59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fe59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fed9d640 10 monclient(hunting): get_auth_request con 0x7f590013e730 auth_method 0 2023-10-30T12:39:51.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.795+0000 7f58fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.796+0000 7f58fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.796+0000 7f58fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.796+0000 7f58fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.797+0000 7f58fe59c640 10 monclient(hunting): handle_auth_done global_id 17744 payload 274 2023-10-30T12:39:51.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.797+0000 7f58fe59c640 1 -- 172.21.15.148:0/2884312825 >> v2:172.21.15.150:3300/0 conn(0x7f590013e730 msgr2=0x7f590013eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:51.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.797+0000 7f58fe59c640 1 --2- 172.21.15.148:0/2884312825 >> v2:172.21.15.150:3300/0 conn(0x7f590013e730 0x7f590013eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.797+0000 7f58fe59c640 1 -- 172.21.15.148:0/2884312825 >> v1:172.21.15.150:6789/0 conn(0x7f5900141940 legacy=0x7f5900141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.797+0000 7f58fe59c640 10 monclient: _finish_hunting 0 2023-10-30T12:39:51.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.797+0000 7f58fe59c640 1 monclient: found mon.noname-b 2023-10-30T12:39:51.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.797+0000 7f58fe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:51.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.797+0000 7f58fe59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:51.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.797+0000 7f58fe59c640 1 -- 172.21.15.148:0/2884312825 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59000aa6a0 con 0x7f59000b7c10 2023-10-30T12:39:51.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.799+0000 7f58fe59c640 1 --2- 172.21.15.148:0/2884312825 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59000b7c10 0x7f59000a8aa0 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009870 tx=0x7f58f002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=427ede51a47142dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 1 -- 172.21.15.148:0/2884312825 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0031640 con 0x7f59000b7c10 2023-10-30T12:39:51.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:51.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 10 monclient: dump: 2023-10-30T12:39:51.813 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.813 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.814 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.814 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.814 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.814 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.815 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.815 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.815 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.815 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 10 monclient: _finish_auth 0 2023-10-30T12:39:51.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 10 monclient: _check_auth_tickets 2023-10-30T12:39:51.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:51.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 1 -- 172.21.15.148:0/2884312825 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f00317b0 con 0x7f59000b7c10 2023-10-30T12:39:51.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f5904e09640 5 monclient: authenticate success, global_id 17744 2023-10-30T12:39:51.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f5904e09640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:39:51.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:51.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f5904e09640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:51.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 1 -- 172.21.15.148:0/2884312825 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0031b50 con 0x7f59000b7c10 2023-10-30T12:39:51.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f5904e09640 4 set_mon_vals no callback set 2023-10-30T12:39:51.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:51.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.802+0000 7f58fdd9b640 10 monclient: dump: 2023-10-30T12:39:51.818 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.818 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.818 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.819 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.819 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.819 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.819 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.819 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.819 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.820 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.803+0000 7f5904e09640 10 monclient: shutdown 2023-10-30T12:39:51.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.803+0000 7f5904e09640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:51.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.803+0000 7f5904e09640 1 -- 172.21.15.148:0/2884312825 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59000b7c10 msgr2=0x7f59000a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.803+0000 7f5904e09640 1 --2- 172.21.15.148:0/2884312825 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59000b7c10 0x7f59000a8aa0 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009870 tx=0x7f58f002b8f0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.803+0000 7f5904e09640 1 -- 172.21.15.148:0/2884312825 shutdown_connections 2023-10-30T12:39:51.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.803+0000 7f5904e09640 1 --2- 172.21.15.148:0/2884312825 >> v2:172.21.15.150:3300/0 conn(0x7f590013e730 0x7f590013eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.803+0000 7f5904e09640 1 --2- 172.21.15.148:0/2884312825 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f59000b7c10 0x7f59000a8aa0 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.803+0000 7f5904e09640 1 -- 172.21.15.148:0/2884312825 >> 172.21.15.148:0/2884312825 conn(0x7f590006c660 msgr2=0x7f59000b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:51.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.803+0000 7f5904e09640 1 -- 172.21.15.148:0/2884312825 shutdown_connections 2023-10-30T12:39:51.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.803+0000 7f5904e09640 1 -- 172.21.15.148:0/2884312825 wait complete. 2023-10-30T12:39:51.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.804+0000 7f5904e09640 10 monclient: build_initial_monmap 2023-10-30T12:39:51.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.804+0000 7f5904e09640 10 monclient: monmap: 2023-10-30T12:39:51.822 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:51.822 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:51.823 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:51.823 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:51.823 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:51.823 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:51.823 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:51.823 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:51.824 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:51.824 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.804+0000 7f5904e09640 1 librados: starting msgr at 2023-10-30T12:39:51.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.804+0000 7f5904e09640 1 librados: starting objecter 2023-10-30T12:39:51.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.804+0000 7f5904e09640 1 Processor -- start 2023-10-30T12:39:51.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.804+0000 7f5904e09640 1 -- start start 2023-10-30T12:39:51.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.804+0000 7f5904e09640 1 librados: setting wanted keys 2023-10-30T12:39:51.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.804+0000 7f5904e09640 1 librados: calling monclient init 2023-10-30T12:39:51.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.804+0000 7f5904e09640 10 monclient: init 2023-10-30T12:39:51.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:51.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:39:51.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59000b7c10 0x7f5900138280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 10 monclient(hunting): picked mon.noname-a con 0x7f59000b7c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:51.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f590013e730 0x7f59001da550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 10 monclient(hunting): picked mon.noname-b con 0x7f590013e730 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:51.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5900141940 0x7f59001cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 10 monclient(hunting): picked mon.noname-c con 0x7f5900141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:51.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59000b2fd0 con 0x7f59000b7c10 2023-10-30T12:39:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59000b1050 con 0x7f590013e730 2023-10-30T12:39:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59001dade0 con 0x7f5900141940 2023-10-30T12:39:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:51.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f5904e09640 10 monclient(hunting): authenticate will time out at 2884.010175s 2023-10-30T12:39:51.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f58fed9d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5900141940 0x7f59001cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f58fe59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59000b7c10 0x7f5900138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f58f5d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f590013e730 0x7f59001da550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f58fed9d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5900141940 0x7f59001cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:51572/0 (socket says 172.21.15.148:51572) 2023-10-30T12:39:51.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f58fed9d640 1 -- 172.21.15.148:0/2386523614 learned_addr learned my addr 172.21.15.148:0/2386523614 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:51.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f58fed9d640 10 monclient(hunting): get_auth_request con 0x7f5900141940 auth_method 0 2023-10-30T12:39:51.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f58fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f58fed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.805+0000 7f58fed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58fe59c640 10 monclient(hunting): get_auth_request con 0x7f59000b7c10 auth_method 0 2023-10-30T12:39:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58fe59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58fe59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58f5d9b640 10 monclient(hunting): get_auth_request con 0x7f590013e730 auth_method 0 2023-10-30T12:39:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58f5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58f5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58f5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58f5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:51.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58f5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:51.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.806+0000 7f58f5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:51.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58f5d9b640 10 monclient(hunting): handle_auth_done global_id 22123 payload 824 2023-10-30T12:39:51.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58f5d9b640 1 -- 172.21.15.148:0/2386523614 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5900141940 msgr2=0x7f59001cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58f5d9b640 1 --2- 172.21.15.148:0/2386523614 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5900141940 0x7f59001cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58f5d9b640 1 -- 172.21.15.148:0/2386523614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59000b7c10 msgr2=0x7f5900138280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58f5d9b640 1 --2- 172.21.15.148:0/2386523614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59000b7c10 0x7f5900138280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58f5d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:39:51.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58f5d9b640 1 monclient: found mon.noname-b 2023-10-30T12:39:51.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:51.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58f5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:39:51.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58f5d9b640 1 -- 172.21.15.148:0/2386523614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59000adcd0 con 0x7f590013e730 2023-10-30T12:39:51.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.809+0000 7f58fed9d640 1 --2- 172.21.15.148:0/2386523614 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f5900141940 0x7f59001cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:39:51.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.811+0000 7f58f5d9b640 1 --2- 172.21.15.148:0/2386523614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f590013e730 0x7f59001da550 secure :-1 s=READY pgs=5450 cs=0 l=1 rev1=1 crypto rx=0x7f58e800c390 tx=0x7f58e800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.811+0000 7f58f77fe640 10 client.?.objecter ms_handle_connect 0x7f590013e730 2023-10-30T12:39:51.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.811+0000 7f58f77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:51.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 1 -- 172.21.15.148:0/2386523614 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e80052d0 con 0x7f590013e730 2023-10-30T12:39:51.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:51.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 10 monclient: dump: 2023-10-30T12:39:51.840 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.840 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.841 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.841 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.841 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.841 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.841 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.841 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.842 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.842 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 10 monclient: _finish_auth 0 2023-10-30T12:39:51.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:39:51.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:51.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 1 -- 172.21.15.148:0/2386523614 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e800e040 con 0x7f590013e730 2023-10-30T12:39:51.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:51.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 1 -- 172.21.15.148:0/2386523614 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e80035f0 con 0x7f590013e730 2023-10-30T12:39:51.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f5904e09640 5 monclient: authenticate success, global_id 22123 2023-10-30T12:39:51.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f5904e09640 10 monclient: _renew_subs 2023-10-30T12:39:51.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f5904e09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:51.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59001daa90 con 0x7f590013e730 2023-10-30T12:39:51.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:51.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:51.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58f77fe640 10 monclient: dump: 2023-10-30T12:39:51.844 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:51.845 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:51.845 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.845 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:51.845 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:51.845 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:51.845 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:51.846 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:51.846 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:51.846 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:51.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f58fd59a640 4 set_mon_vals no callback set 2023-10-30T12:39:51.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f5904e09640 10 client.22123.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:51.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f5904e09640 10 monclient: _renew_subs 2023-10-30T12:39:51.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f5904e09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:51.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59001ce540 con 0x7f590013e730 2023-10-30T12:39:51.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.815+0000 7f5904e09640 1 librados: init done 2023-10-30T12:39:51.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.816+0000 7f5904e09640 10 client.22123.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:51.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.816+0000 7f5904e09640 15 client.22123.objecter _session_command_op_assign -1 1 2023-10-30T12:39:51.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.816+0000 7f5904e09640 20 client.22123.objecter _get_session osd=-1 returning homeless 2023-10-30T12:39:51.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.816+0000 7f5904e09640 10 client.22123.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:51.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.816+0000 7f5904e09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:39:51.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.816+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5900142220 con 0x7f590013e730 2023-10-30T12:39:51.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.833+0000 7f58f77fe640 1 -- 172.21.15.148:0/2386523614 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f58e8003760 con 0x7f590013e730 2023-10-30T12:39:51.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.834+0000 7f58f77fe640 1 --2- 172.21.15.148:0/2386523614 >> v2:172.21.15.148:6800/3205090179 conn(0x7f58e006c810 0x7f58e006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.834+0000 7f58fe59c640 1 --2- 172.21.15.148:0/2386523614 >> v2:172.21.15.148:6800/3205090179 conn(0x7f58e006c810 0x7f58e006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.834+0000 7f58fe59c640 10 monclient: get_auth_request con 0x7f58e006c810 auth_method 0 2023-10-30T12:39:51.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.834+0000 7f58fe59c640 1 --2- 172.21.15.148:0/2386523614 >> v2:172.21.15.148:6800/3205090179 conn(0x7f58e006c810 0x7f58e006ecd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009870 tx=0x7f58f003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.834+0000 7f58f77fe640 10 client.22123.objecter ms_handle_connect 0x7f58e006c810 2023-10-30T12:39:51.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 1 -- 172.21.15.148:0/2386523614 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f58e8057980 con 0x7f590013e730 2023-10-30T12:39:51.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 10 client.22123.objecter ms_dispatch 0x7f5900000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:51.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 3 client.22123.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:51.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 3 client.22123.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:51.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 10 client.22123.objecter checking command 1 2023-10-30T12:39:51.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 1 --2- 172.21.15.148:0/2386523614 >> v2:172.21.15.150:6804/4190376848 conn(0x7f58e0073b30 0x7f58e0075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:51.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 20 client.22123.objecter _get_session s=0x7f58e80035f0 osd=5 3 2023-10-30T12:39:51.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 20 client.22123.objecter put_session s=0x7f58e80035f0 osd=5 3 2023-10-30T12:39:51.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 15 client.22123.objecter _session_command_op_remove -1 1 2023-10-30T12:39:51.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 20 client.22123.objecter _get_session s=0x7f58e80035f0 osd=5 3 2023-10-30T12:39:51.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 20 client.22123.objecter get_session s=0x7f58e80035f0 osd=5 3 2023-10-30T12:39:51.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 15 client.22123.objecter _session_command_op_assign 5 1 2023-10-30T12:39:51.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 20 client.22123.objecter put_session s=0x7f58e80035f0 osd=5 4 2023-10-30T12:39:51.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 10 client.22123.objecter _send_command 1 2023-10-30T12:39:51.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 1 -- 172.21.15.148:0/2386523614 --> v2:172.21.15.150:6804/4190376848 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58e80052d0 con 0x7f58e0073b30 2023-10-30T12:39:51.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.836+0000 7f58f77fe640 20 client.22123.objecter dump_active .. 0 homeless 2023-10-30T12:39:51.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.837+0000 7f58f77fe640 1 -- 172.21.15.148:0/2386523614 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59000b1050 con 0x7f590013e730 2023-10-30T12:39:51.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.837+0000 7f58f77fe640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:39:51.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.837+0000 7f58fed9d640 1 --2- 172.21.15.148:0/2386523614 >> v2:172.21.15.150:6804/4190376848 conn(0x7f58e0073b30 0x7f58e0075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:51.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.837+0000 7f58fed9d640 10 monclient: get_auth_request con 0x7f58e0073b30 auth_method 0 2023-10-30T12:39:51.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.840+0000 7f58f77fe640 10 client.22123.objecter ms_handle_connect 0x7f58e0073b30 2023-10-30T12:39:51.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.840+0000 7f58fed9d640 1 --2- 172.21.15.148:0/2386523614 >> v2:172.21.15.150:6804/4190376848 conn(0x7f58e0073b30 0x7f58e0075fd0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:51.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.863+0000 7f58f77fe640 1 -- 172.21.15.148:0/2386523614 <== osd.5 v2:172.21.15.150:6804/4190376848 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f58e80052d0 con 0x7f58e0073b30 2023-10-30T12:39:51.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.863+0000 7f58f77fe640 10 client.22123.objecter ms_dispatch 0x7f5900000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:39:51.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.863+0000 7f58f77fe640 10 client.22123.objecter _finish_command 1 = system:0 2023-10-30T12:39:51.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.863+0000 7f58f77fe640 20 client.22123.objecter put_session s=0x7f58e80035f0 osd=5 4 2023-10-30T12:39:51.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.863+0000 7f58f77fe640 15 client.22123.objecter _session_command_op_remove 5 1 2023-10-30T12:39:51.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 10 client.22123.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:39:51.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 15 client.22123.objecter _session_command_op_assign -1 2 2023-10-30T12:39:51.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 20 client.22123.objecter _get_session s=0x7f58e80035f0 osd=5 3 2023-10-30T12:39:51.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 20 client.22123.objecter put_session s=0x7f58e80035f0 osd=5 3 2023-10-30T12:39:51.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 20 client.22123.objecter _get_session s=0x7f58e80035f0 osd=5 3 2023-10-30T12:39:51.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 15 client.22123.objecter _session_command_op_remove -1 2 2023-10-30T12:39:51.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 20 client.22123.objecter get_session s=0x7f58e80035f0 osd=5 3 2023-10-30T12:39:51.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 15 client.22123.objecter _session_command_op_assign 5 2 2023-10-30T12:39:51.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 20 client.22123.objecter put_session s=0x7f58e80035f0 osd=5 4 2023-10-30T12:39:51.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 10 client.22123.objecter _send_command 2 2023-10-30T12:39:51.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.875+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 --> v2:172.21.15.150:6804/4190376848 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f59000a8fe0 con 0x7f58e0073b30 2023-10-30T12:39:51.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.889+0000 7f58f77fe640 1 -- 172.21.15.148:0/2386523614 <== osd.5 v2:172.21.15.150:6804/4190376848 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f58f800f040 con 0x7f58e0073b30 2023-10-30T12:39:51.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.889+0000 7f58f77fe640 10 client.22123.objecter ms_dispatch 0x7f5900000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:39:51.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.889+0000 7f58f77fe640 10 client.22123.objecter _finish_command 2 = system:0 2023-10-30T12:39:51.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.889+0000 7f58f77fe640 20 client.22123.objecter put_session s=0x7f58e80035f0 osd=5 4 2023-10-30T12:39:51.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.889+0000 7f58f77fe640 15 client.22123.objecter _session_command_op_remove 5 2 2023-10-30T12:39:51.892 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T12:39:51.892 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:39:51.893 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T12:39:51.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.889+0000 7f5904e09640 10 librados: watch_flush enter 2023-10-30T12:39:51.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 10 librados: watch_flush exit 2023-10-30T12:39:51.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 10 client.22123.objecter close_session for osd.5 2023-10-30T12:39:51.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 >> v2:172.21.15.150:6804/4190376848 conn(0x7f58e0073b30 msgr2=0x7f58e0075fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 --2- 172.21.15.148:0/2386523614 >> v2:172.21.15.150:6804/4190376848 conn(0x7f58e0073b30 0x7f58e0075fd0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:51.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 20 client.22123.objecter put_session s=0x7f58e80035f0 osd=5 1 2023-10-30T12:39:51.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 20 client.22123.objecter shutdown clearing up homeless session... 2023-10-30T12:39:51.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 10 client.22123.objecter successfully canceled tick 2023-10-30T12:39:51.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 >> v2:172.21.15.148:6800/3205090179 conn(0x7f58e006c810 msgr2=0x7f58e006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 --2- 172.21.15.148:0/2386523614 >> v2:172.21.15.148:6800/3205090179 conn(0x7f58e006c810 0x7f58e006ecd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009870 tx=0x7f58f003c040 comp rx=0 tx=0).stop 2023-10-30T12:39:51.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 10 monclient: shutdown 2023-10-30T12:39:51.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:51.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f590013e730 msgr2=0x7f59001da550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:51.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 --2- 172.21.15.148:0/2386523614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f590013e730 0x7f59001da550 secure :-1 s=READY pgs=5450 cs=0 l=1 rev1=1 crypto rx=0x7f58e800c390 tx=0x7f58e800c850 comp rx=0 tx=0).stop 2023-10-30T12:39:51.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f58fed9d640 1 -- 172.21.15.148:0/2386523614 reap_dead start 2023-10-30T12:39:51.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 shutdown_connections 2023-10-30T12:39:51.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 >> 172.21.15.148:0/2386523614 conn(0x7f590006c660 msgr2=0x7f590013c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:51.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 shutdown_connections 2023-10-30T12:39:51.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 -- 172.21.15.148:0/2386523614 wait complete. 2023-10-30T12:39:51.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.890+0000 7f5904e09640 1 librados: shutdown 2023-10-30T12:39:51.902 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T12:39:52.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.999+0000 7f714fac2640 10 monclient: get_monmap_and_config 2023-10-30T12:39:52.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.999+0000 7f714fac2640 10 monclient: build_initial_monmap 2023-10-30T12:39:52.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:51.999+0000 7f714fac2640 10 monclient: monmap: 2023-10-30T12:39:52.002 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:52.002 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.002 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:52.000907+0000 2023-10-30T12:39:52.002 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:52.000907+0000 2023-10-30T12:39:52.003 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:52.003 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.003 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:52.003 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:52.003 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:52.004 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:52.004 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:52.004 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.000+0000 7f714fac2640 1 Processor -- start 2023-10-30T12:39:52.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.000+0000 7f714fac2640 1 -- start start 2023-10-30T12:39:52.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.000+0000 7f714fac2640 10 monclient: init 2023-10-30T12:39:52.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:52.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-10-30T12:39:52.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f71480ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 10 monclient(hunting): picked mon.noname-b con 0x7f71480aeb40 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:52.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f71480aabc0 0x7f71480acde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 10 monclient(hunting): picked mon.noname-c con 0x7f71480aabc0 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:52.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71480ab040 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:52.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71480ac130 con 0x7f71480ab040 2023-10-30T12:39:52.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7148067340 con 0x7f71480aabc0 2023-10-30T12:39:52.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71480aef20 con 0x7f71480aeb40 2023-10-30T12:39:52.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:52.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714fac2640 10 monclient(hunting): authenticate will time out at 2884.206133s 2023-10-30T12:39:52.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714d85e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f71480ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.001+0000 7f714d05d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f71480aabc0 0x7f71480acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d85e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f71480ac8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:51578/0 (socket says 172.21.15.148:51578) 2023-10-30T12:39:52.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d85e640 1 -- 172.21.15.148:0/1116670438 learned_addr learned my addr 172.21.15.148:0/1116670438 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:52.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d85e640 10 monclient(hunting): get_auth_request con 0x7f71480aeb40 auth_method 0 2023-10-30T12:39:52.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d85e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:52.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d85e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:52.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d85e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:52.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d05d640 10 monclient(hunting): get_auth_request con 0x7f71480aabc0 auth_method 0 2023-10-30T12:39:52.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d05d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:52.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d05d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:52.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d05d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:52.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d85e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:52.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d85e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:52.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.002+0000 7f714d85e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:52.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.003+0000 7f714d05d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:52.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.003+0000 7f714d05d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:52.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.003+0000 7f714d05d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:52.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.004+0000 7f714d85e640 10 monclient(hunting): handle_auth_done global_id 17750 payload 274 2023-10-30T12:39:52.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.004+0000 7f714d85e640 1 -- 172.21.15.148:0/1116670438 >> v2:172.21.15.150:3300/0 conn(0x7f71480aabc0 msgr2=0x7f71480acde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:52.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.004+0000 7f714d85e640 1 --2- 172.21.15.148:0/1116670438 >> v2:172.21.15.150:3300/0 conn(0x7f71480aabc0 0x7f71480acde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.004+0000 7f714d85e640 1 -- 172.21.15.148:0/1116670438 >> v1:172.21.15.150:6789/0 conn(0x7f71480ab040 legacy=0x7f714813b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.004+0000 7f714d85e640 10 monclient: _finish_hunting 0 2023-10-30T12:39:52.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.004+0000 7f714d85e640 1 monclient: found mon.noname-b 2023-10-30T12:39:52.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.004+0000 7f714d85e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:52.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.004+0000 7f714d85e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:52.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.004+0000 7f714d85e640 1 -- 172.21.15.148:0/1116670438 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71480aa6a0 con 0x7f71480aeb40 2023-10-30T12:39:52.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.006+0000 7f714d85e640 1 --2- 172.21.15.148:0/1116670438 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f71480ac8a0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f7144009e50 tx=0x7f714402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=329606c764d6c35b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:52.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 1 -- 172.21.15.148:0/1116670438 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714402c9e0 con 0x7f71480aeb40 2023-10-30T12:39:52.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:52.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:52.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 10 monclient: dump: 2023-10-30T12:39:52.021 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:52.021 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.021 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.021 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.021 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:52.022 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.022 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:52.022 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:52.022 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:52.022 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 10 monclient: _finish_auth 0 2023-10-30T12:39:52.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 10 monclient: _check_auth_tickets 2023-10-30T12:39:52.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:52.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 1 -- 172.21.15.148:0/1116670438 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714402cb50 con 0x7f71480aeb40 2023-10-30T12:39:52.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714fac2640 5 monclient: authenticate success, global_id 17750 2023-10-30T12:39:52.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714fac2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:39:52.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:52.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714fac2640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:52.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714fac2640 4 set_mon_vals no callback set 2023-10-30T12:39:52.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 1 -- 172.21.15.148:0/1116670438 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71440343f0 con 0x7f71480aeb40 2023-10-30T12:39:52.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:52.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:52.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.009+0000 7f714c85c640 10 monclient: dump: 2023-10-30T12:39:52.025 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:52.026 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.026 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.026 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.026 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:52.027 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.027 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:52.027 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:52.027 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:52.027 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.010+0000 7f714fac2640 10 monclient: shutdown 2023-10-30T12:39:52.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.010+0000 7f714fac2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:52.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.010+0000 7f714fac2640 1 -- 172.21.15.148:0/1116670438 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 msgr2=0x7f71480ac8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.010+0000 7f714fac2640 1 --2- 172.21.15.148:0/1116670438 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f71480ac8a0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f7144009e50 tx=0x7f714402be40 comp rx=0 tx=0).stop 2023-10-30T12:39:52.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.010+0000 7f714fac2640 1 -- 172.21.15.148:0/1116670438 shutdown_connections 2023-10-30T12:39:52.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.010+0000 7f714fac2640 1 --2- 172.21.15.148:0/1116670438 >> v2:172.21.15.150:3300/0 conn(0x7f71480aabc0 0x7f71480acde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.010+0000 7f714fac2640 1 --2- 172.21.15.148:0/1116670438 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f71480ac8a0 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.010+0000 7f714fac2640 1 -- 172.21.15.148:0/1116670438 >> 172.21.15.148:0/1116670438 conn(0x7f71480b3260 msgr2=0x7f71480b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:52.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.010+0000 7f714fac2640 1 -- 172.21.15.148:0/1116670438 shutdown_connections 2023-10-30T12:39:52.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.010+0000 7f714fac2640 1 -- 172.21.15.148:0/1116670438 wait complete. 2023-10-30T12:39:52.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.011+0000 7f714fac2640 10 monclient: build_initial_monmap 2023-10-30T12:39:52.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.011+0000 7f714fac2640 10 monclient: monmap: 2023-10-30T12:39:52.030 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:52.030 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:52.030 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:52.031 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:52.031 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:52.031 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:52.031 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:52.031 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:52.032 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:52.032 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.011+0000 7f714fac2640 1 librados: starting msgr at 2023-10-30T12:39:52.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.011+0000 7f714fac2640 1 librados: starting objecter 2023-10-30T12:39:52.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.011+0000 7f714fac2640 1 Processor -- start 2023-10-30T12:39:52.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.011+0000 7f714fac2640 1 -- start start 2023-10-30T12:39:52.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.011+0000 7f714fac2640 1 librados: setting wanted keys 2023-10-30T12:39:52.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.011+0000 7f714fac2640 1 librados: calling monclient init 2023-10-30T12:39:52.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.011+0000 7f714fac2640 10 monclient: init 2023-10-30T12:39:52.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:52.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:39:52.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71480aabc0 0x7f71481379f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 10 monclient(hunting): picked mon.noname-a con 0x7f71480aabc0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:52.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71480ab040 0x7f7148135580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 10 monclient(hunting): picked mon.noname-b con 0x7f71480ab040 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:52.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f7148135ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 10 monclient(hunting): picked mon.noname-c con 0x7f71480aeb40 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:52.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7148071880 con 0x7f71480aabc0 2023-10-30T12:39:52.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71480b2550 con 0x7f71480ab040 2023-10-30T12:39:52.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71480b20c0 con 0x7f71480aeb40 2023-10-30T12:39:52.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:52.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714fac2640 10 monclient(hunting): authenticate will time out at 2884.217340s 2023-10-30T12:39:52.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714e05f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f7148135ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.012+0000 7f714d85e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71480aabc0 0x7f71481379f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d05d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71480ab040 0x7f7148135580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714e05f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f7148135ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:51590/0 (socket says 172.21.15.148:51590) 2023-10-30T12:39:52.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714e05f640 1 -- 172.21.15.148:0/1148056900 learned_addr learned my addr 172.21.15.148:0/1148056900 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:52.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714e05f640 10 monclient(hunting): get_auth_request con 0x7f71480aeb40 auth_method 0 2023-10-30T12:39:52.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714e05f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:52.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714e05f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:52.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714e05f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:52.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d85e640 10 monclient(hunting): get_auth_request con 0x7f71480aabc0 auth_method 0 2023-10-30T12:39:52.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d85e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:52.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d85e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:52.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d85e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:52.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d05d640 10 monclient(hunting): get_auth_request con 0x7f71480ab040 auth_method 0 2023-10-30T12:39:52.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d05d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:52.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d05d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:52.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d05d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:52.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714e05f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:52.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714e05f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:52.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714e05f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:52.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d85e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:52.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d85e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:52.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d85e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:52.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d05d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:52.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d05d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:52.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.013+0000 7f714d05d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:52.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.016+0000 7f714e05f640 10 monclient(hunting): handle_auth_done global_id 17753 payload 824 2023-10-30T12:39:52.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.016+0000 7f714e05f640 1 -- 172.21.15.148:0/1148056900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71480ab040 msgr2=0x7f7148135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.016+0000 7f714e05f640 1 --2- 172.21.15.148:0/1148056900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71480ab040 0x7f7148135580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.016+0000 7f714e05f640 1 -- 172.21.15.148:0/1148056900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71480aabc0 msgr2=0x7f71481379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.016+0000 7f714e05f640 1 --2- 172.21.15.148:0/1148056900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71480aabc0 0x7f71481379f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.016+0000 7f714e05f640 10 monclient: _finish_hunting 0 2023-10-30T12:39:52.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.016+0000 7f714e05f640 1 monclient: found mon.noname-c 2023-10-30T12:39:52.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.016+0000 7f714e05f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:52.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.016+0000 7f714e05f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:39:52.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.016+0000 7f714e05f640 1 -- 172.21.15.148:0/1148056900 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7148136f80 con 0x7f71480aeb40 2023-10-30T12:39:52.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.018+0000 7f714e05f640 1 --2- 172.21.15.148:0/1148056900 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f7148135ac0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f714000c3a0 tx=0x7f714000c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:52.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.018+0000 7f713e7fc640 10 client.?.objecter ms_handle_connect 0x7f71480aeb40 2023-10-30T12:39:52.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.018+0000 7f713e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:52.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 1 -- 172.21.15.148:0/1148056900 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71400052d0 con 0x7f71480aeb40 2023-10-30T12:39:52.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:52.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:52.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 10 monclient: dump: 2023-10-30T12:39:52.047 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:52.047 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.047 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.047 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.047 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:52.048 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.048 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:52.048 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:52.048 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:52.048 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:39:52.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:39:52.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:52.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 1 -- 172.21.15.148:0/1148056900 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714000e040 con 0x7f71480aeb40 2023-10-30T12:39:52.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f714fac2640 5 monclient: authenticate success, global_id 17753 2023-10-30T12:39:52.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f714fac2640 10 monclient: _renew_subs 2023-10-30T12:39:52.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f714fac2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:52.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f714fac2640 1 -- 172.21.15.148:0/1148056900 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7148137260 con 0x7f71480aeb40 2023-10-30T12:39:52.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:52.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 1 -- 172.21.15.148:0/1148056900 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71400035f0 con 0x7f71480aeb40 2023-10-30T12:39:52.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:52.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:52.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713e7fc640 10 monclient: dump: 2023-10-30T12:39:52.051 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:52.051 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.051 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.052 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.052 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:52.052 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.052 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:52.052 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:52.053 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:52.053 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f713ffff640 4 set_mon_vals no callback set 2023-10-30T12:39:52.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f714fac2640 10 client.17753.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:52.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f714fac2640 10 monclient: _renew_subs 2023-10-30T12:39:52.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f714fac2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:52.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f714fac2640 1 -- 172.21.15.148:0/1148056900 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7148134c00 con 0x7f71480aeb40 2023-10-30T12:39:52.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.022+0000 7f714fac2640 1 librados: init done 2023-10-30T12:39:52.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.023+0000 7f714fac2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:52.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.023+0000 7f714fac2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:52.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.023+0000 7f714fac2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:52.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.023+0000 7f714fac2640 1 -- 172.21.15.148:0/1148056900 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7148134e30 con 0x7f71480aeb40 2023-10-30T12:39:52.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.042+0000 7f713e7fc640 1 -- 172.21.15.148:0/1148056900 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f7140003760 con 0x7f71480aeb40 2023-10-30T12:39:52.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.042+0000 7f713e7fc640 1 --2- 172.21.15.148:0/1148056900 >> v2:172.21.15.148:6800/3205090179 conn(0x7f712c06c7c0 0x7f712c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.042+0000 7f714d85e640 1 --2- 172.21.15.148:0/1148056900 >> v2:172.21.15.148:6800/3205090179 conn(0x7f712c06c7c0 0x7f712c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.042+0000 7f714d85e640 10 monclient: get_auth_request con 0x7f712c06c7c0 auth_method 0 2023-10-30T12:39:52.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.043+0000 7f713e7fc640 10 client.17753.objecter ms_handle_connect 0x7f712c06c7c0 2023-10-30T12:39:52.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.043+0000 7f714d85e640 1 --2- 172.21.15.148:0/1148056900 >> v2:172.21.15.148:6800/3205090179 conn(0x7f712c06c7c0 0x7f712c06ec80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7144035000 tx=0x7f7144030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:52.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.293+0000 7f713e7fc640 1 -- 172.21.15.148:0/1148056900 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(900..900 src has 501..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f7140057590 con 0x7f71480aeb40 2023-10-30T12:39:52.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.293+0000 7f713e7fc640 10 client.17753.objecter ms_dispatch 0x7f7148000b90 osd_map(900..900 src has 501..900) v4 2023-10-30T12:39:52.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.293+0000 7f713e7fc640 3 client.17753.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:52.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.293+0000 7f713e7fc640 3 client.17753.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:52.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.293+0000 7f713e7fc640 20 client.17753.objecter dump_active .. 0 homeless 2023-10-30T12:39:52.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.293+0000 7f713e7fc640 1 -- 172.21.15.148:0/1148056900 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f714005b630 con 0x7f71480aeb40 2023-10-30T12:39:52.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.293+0000 7f713e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:52.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.293+0000 7f713e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:52.429 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:07.503 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:52.433 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:07.507 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:52.449 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:07.523 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:52.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.475+0000 7f714fac2640 1 -- 172.21.15.148:0/1148056900 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f7148136f80 con 0x7f712c06c7c0 2023-10-30T12:39:52.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.475+0000 7f713e7fc640 1 -- 172.21.15.148:0/1148056900 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7148136f80 con 0x7f712c06c7c0 2023-10-30T12:39:52.477 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T12:39:52.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 10 librados: watch_flush enter 2023-10-30T12:39:52.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 10 librados: watch_flush exit 2023-10-30T12:39:52.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 20 client.17753.objecter shutdown clearing up homeless session... 2023-10-30T12:39:52.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 10 client.17753.objecter successfully canceled tick 2023-10-30T12:39:52.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 -- 172.21.15.148:0/1148056900 >> v2:172.21.15.148:6800/3205090179 conn(0x7f712c06c7c0 msgr2=0x7f712c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 --2- 172.21.15.148:0/1148056900 >> v2:172.21.15.148:6800/3205090179 conn(0x7f712c06c7c0 0x7f712c06ec80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7144035000 tx=0x7f7144030000 comp rx=0 tx=0).stop 2023-10-30T12:39:52.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 10 monclient: shutdown 2023-10-30T12:39:52.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:52.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 -- 172.21.15.148:0/1148056900 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 msgr2=0x7f7148135ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 --2- 172.21.15.148:0/1148056900 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f7148135ac0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f714000c3a0 tx=0x7f714000c860 comp rx=0 tx=0).stop 2023-10-30T12:39:52.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 -- 172.21.15.148:0/1148056900 shutdown_connections 2023-10-30T12:39:52.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 --2- 172.21.15.148:0/1148056900 >> v2:172.21.15.148:6800/3205090179 conn(0x7f712c06c7c0 0x7f712c06ec80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 --2- 172.21.15.148:0/1148056900 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f71480aeb40 0x7f7148135ac0 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 --2- 172.21.15.148:0/1148056900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71480ab040 0x7f7148135580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 --2- 172.21.15.148:0/1148056900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71480aabc0 0x7f71481379f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 -- 172.21.15.148:0/1148056900 >> 172.21.15.148:0/1148056900 conn(0x7f71480b3260 msgr2=0x7f71480ad6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:52.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 -- 172.21.15.148:0/1148056900 shutdown_connections 2023-10-30T12:39:52.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 -- 172.21.15.148:0/1148056900 wait complete. 2023-10-30T12:39:52.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.477+0000 7f714fac2640 1 librados: shutdown 2023-10-30T12:39:52.483 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:07.527 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:52.483 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:07.529 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:52.490 INFO:tasks.ceph:Scrubbing osd.6 2023-10-30T12:39:52.490 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:39:52.493 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:07.567 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:52.495 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:07.569 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:52.498 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:07.546 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:52.513 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:07.586 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:52.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.595+0000 7f484ee51640 10 monclient: get_monmap_and_config 2023-10-30T12:39:52.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.595+0000 7f484ee51640 10 monclient: build_initial_monmap 2023-10-30T12:39:52.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.595+0000 7f484ee51640 10 monclient: monmap: 2023-10-30T12:39:52.598 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:52.598 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.598 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:52.596769+0000 2023-10-30T12:39:52.598 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:52.596769+0000 2023-10-30T12:39:52.599 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:52.599 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.599 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:52.599 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:52.599 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:52.600 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:52.600 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:52.600 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.596+0000 7f484ee51640 1 Processor -- start 2023-10-30T12:39:52.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.596+0000 7f484ee51640 1 -- start start 2023-10-30T12:39:52.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.596+0000 7f484ee51640 10 monclient: init 2023-10-30T12:39:52.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:52.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T12:39:52.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48480b7c10 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:52.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4848141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:52.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f484814a460 0x7f484814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 10 monclient(hunting): picked mon.noname-b con 0x7f484814a460 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:52.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480a9350 con 0x7f48480b7c10 2023-10-30T12:39:52.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480aa6a0 con 0x7f4848141940 2023-10-30T12:39:52.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48480670a0 con 0x7f484814a460 2023-10-30T12:39:52.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:52.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.597+0000 7f484ee51640 10 monclient(hunting): authenticate will time out at 2884.802301s 2023-10-30T12:39:52.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.600+0000 7f484cbed640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f48480b7c10 0x7f48480a8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55836/0 (socket says 172.21.15.148:55836) 2023-10-30T12:39:52.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.600+0000 7f484cbed640 1 -- 172.21.15.148:0/3800124286 learned_addr learned my addr 172.21.15.148:0/3800124286 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:39:52.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.632+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057986600 0 0) 0x7f483c001110 con 0x7f48480b7c10 2023-10-30T12:39:52.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.632+0000 7f48477fe640 10 monclient(hunting): _negotiate 2023-10-30T12:39:52.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.632+0000 7f48477fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:52.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.632+0000 7f48477fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:52.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.632+0000 7f48477fe640 10 monclient(hunting): my global_id is 16302 2023-10-30T12:39:52.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.632+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4838003520 con 0x7f48480b7c10 2023-10-30T12:39:52.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 980930838 0 0) 0x7f483c001510 con 0x7f48480b7c10 2023-10-30T12:39:52.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f484814a460 msgr2=0x7f484814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:52.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 --2- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f484814a460 0x7f484814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 >> v1:172.21.15.150:6789/0 conn(0x7f4848141940 legacy=0x7f4848141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:52.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient: _finish_hunting 0 2023-10-30T12:39:52.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:39:52.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:52.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.148:6789/0 2023-10-30T12:39:52.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48480aa910 con 0x7f48480b7c10 2023-10-30T12:39:52.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient: _finish_auth 0 2023-10-30T12:39:52.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient: _check_auth_tickets 2023-10-30T12:39:52.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:52.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1567732308 0 0) 0x7f483c0019b0 con 0x7f48480b7c10 2023-10-30T12:39:52.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:52.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:39:52.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient: dump: 2023-10-30T12:39:52.651 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:52.651 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.651 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.651 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.651 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:52.651 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.652 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:52.652 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:52.652 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:52.652 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] but i'm connected to v1:172.21.15.148:6789/0, reconnecting 2023-10-30T12:39:52.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:52.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 >> v1:172.21.15.148:6789/0 conn(0x7f48480b7c10 legacy=0x7f48480a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:39:52.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 --2- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f484814a460 0x7f48380039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient(hunting): picked mon.c con 0x7f484814a460 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:52.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 --2- 172.21.15.148:0/3800124286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48380040e0 0x7f48380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient(hunting): picked mon.b con 0x7f48380040e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:52.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 1 --2- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4838004e20 0x7f483800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.645+0000 7f48477fe640 10 monclient(hunting): picked mon.a con 0x7f4838004e20 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:52.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.646+0000 7f48477fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.646+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48480670a0 con 0x7f4838004e20 2023-10-30T12:39:52.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.646+0000 7f48477fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.646+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4838005640 con 0x7f48380040e0 2023-10-30T12:39:52.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.646+0000 7f48477fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.646+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483800bec0 con 0x7f484814a460 2023-10-30T12:39:52.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.646+0000 7f48477fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:52.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.646+0000 7f48477fe640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:39:52.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.657+0000 7f484cbed640 1 --2- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f484814a460 0x7f48380039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.664+0000 7f484cbed640 10 monclient(hunting): get_auth_request con 0x7f484814a460 auth_method 0 2023-10-30T12:39:52.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.664+0000 7f484cbed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:52.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.664+0000 7f484cbed640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:52.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.664+0000 7f484cbed640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:39:52.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.665+0000 7f484d3ee640 1 --2- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4838004e20 0x7f483800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.666+0000 7f484d3ee640 10 monclient(hunting): get_auth_request con 0x7f4838004e20 auth_method 0 2023-10-30T12:39:52.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.666+0000 7f484d3ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:52.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.666+0000 7f484d3ee640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:52.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.666+0000 7f484d3ee640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:39:52.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.666+0000 7f484cbed640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:52.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.666+0000 7f484cbed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:52.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.666+0000 7f484cbed640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:39:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.675+0000 7f484d3ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.675+0000 7f484d3ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.675+0000 7f484d3ee640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:39:52.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 10 monclient(hunting): handle_auth_done global_id 16302 payload 293 2023-10-30T12:39:52.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 1 -- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f484814a460 msgr2=0x7f48380039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:52.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 1 --2- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f484814a460 0x7f48380039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 1 -- 172.21.15.148:0/3800124286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48380040e0 msgr2=0x7f48380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:52.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 1 --2- 172.21.15.148:0/3800124286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48380040e0 0x7f48380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 10 monclient: _finish_hunting 0 2023-10-30T12:39:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 1 monclient: found mon.a 2023-10-30T12:39:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:39:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 1 -- 172.21.15.148:0/3800124286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c001510 con 0x7f4838004e20 2023-10-30T12:39:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 10 monclient: _finish_auth 0 2023-10-30T12:39:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 10 monclient: _check_auth_tickets 2023-10-30T12:39:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484d3ee640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484ee51640 5 monclient: authenticate success, global_id 16302 2023-10-30T12:39:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.679+0000 7f484ee51640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:39:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.683+0000 7f484d3ee640 1 --2- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4838004e20 0x7f483800b980 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7f4840009e40 tx=0x7f48380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec561acf609fb30a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484002cd10 con 0x7f4838004e20 2023-10-30T12:39:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f48477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f48477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:39:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f48477fe640 10 monclient: dump: 2023-10-30T12:39:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:52.697 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4840034440 con 0x7f4838004e20 2023-10-30T12:39:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f484ee51640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:39:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f48477fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f48477fe640 1 -- 172.21.15.148:0/3800124286 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4840034800 con 0x7f4838004e20 2023-10-30T12:39:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f484ee51640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f484ee51640 4 set_mon_vals no callback set 2023-10-30T12:39:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f48477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f48477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:39:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.693+0000 7f48477fe640 10 monclient: dump: 2023-10-30T12:39:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:52.700 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.700 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.700 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.700 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:52.700 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.701 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:52.701 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:52.701 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:52.701 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.694+0000 7f484ee51640 10 monclient: shutdown 2023-10-30T12:39:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.694+0000 7f484ee51640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:52.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.694+0000 7f484ee51640 1 -- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4838004e20 msgr2=0x7f483800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.694+0000 7f484ee51640 1 --2- 172.21.15.148:0/3800124286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4838004e20 0x7f483800b980 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7f4840009e40 tx=0x7f48380057f0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.694+0000 7f484d3ee640 1 -- 172.21.15.148:0/3800124286 reap_dead start 2023-10-30T12:39:52.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.694+0000 7f484ee51640 1 -- 172.21.15.148:0/3800124286 shutdown_connections 2023-10-30T12:39:52.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.694+0000 7f484ee51640 1 -- 172.21.15.148:0/3800124286 >> 172.21.15.148:0/3800124286 conn(0x7f484806c660 msgr2=0x7f48480b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:52.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.694+0000 7f484ee51640 1 -- 172.21.15.148:0/3800124286 shutdown_connections 2023-10-30T12:39:52.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.694+0000 7f484ee51640 1 -- 172.21.15.148:0/3800124286 wait complete. 2023-10-30T12:39:52.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.695+0000 7f484ee51640 10 monclient: build_initial_monmap 2023-10-30T12:39:52.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.695+0000 7f484ee51640 10 monclient: monmap: 2023-10-30T12:39:52.705 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:52.705 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:52.706 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:52.706 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:52.706 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:52.706 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:52.706 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:52.707 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:52.707 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:52.707 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 librados: starting msgr at 2023-10-30T12:39:52.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 librados: starting objecter 2023-10-30T12:39:52.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 Processor -- start 2023-10-30T12:39:52.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 -- start start 2023-10-30T12:39:52.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 librados: setting wanted keys 2023-10-30T12:39:52.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 librados: calling monclient init 2023-10-30T12:39:52.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient: init 2023-10-30T12:39:52.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:52.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:39:52.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48480acc20 0x7f48481da7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient(hunting): picked mon.noname-b con 0x7f48480acc20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:52.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48481dad10 0x7f4848136370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient(hunting): picked mon.noname-a con 0x7f48481dad10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:52.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f48480ad000 0x7f48481368b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient(hunting): picked mon.noname-c con 0x7f48480ad000 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:52.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48480b33c0 con 0x7f48481dad10 2023-10-30T12:39:52.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48481da0a0 con 0x7f48480acc20 2023-10-30T12:39:52.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:52.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48481da2b0 con 0x7f48480ad000 2023-10-30T12:39:52.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:52.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.696+0000 7f484ee51640 10 monclient(hunting): authenticate will time out at 2884.901796s 2023-10-30T12:39:52.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.699+0000 7f4847fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48481dad10 0x7f4848136370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.699+0000 7f484d3ee640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f48480ad000 0x7f48481368b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.701+0000 7f4847fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48481dad10 0x7f4848136370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52350/0 (socket says 172.21.15.148:52350) 2023-10-30T12:39:52.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.701+0000 7f4847fff640 1 -- 172.21.15.148:0/3133947489 learned_addr learned my addr 172.21.15.148:0/3133947489 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:52.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.701+0000 7f4847fff640 10 monclient(hunting): get_auth_request con 0x7f48481dad10 auth_method 0 2023-10-30T12:39:52.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.701+0000 7f4847fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:52.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.701+0000 7f4847fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:52.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.701+0000 7f4847fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:52.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.703+0000 7f4847fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:52.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.703+0000 7f4847fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:52.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.703+0000 7f4847fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:52.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.703+0000 7f484d3ee640 10 monclient(hunting): get_auth_request con 0x7f48480ad000 auth_method 0 2023-10-30T12:39:52.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.703+0000 7f484d3ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:52.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.703+0000 7f484d3ee640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:52.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.703+0000 7f484d3ee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:52.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.710+0000 7f4847fff640 10 monclient(hunting): handle_auth_done global_id 16305 payload 824 2023-10-30T12:39:52.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.710+0000 7f4847fff640 1 -- 172.21.15.148:0/3133947489 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f48480ad000 msgr2=0x7f48481368b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.710+0000 7f4847fff640 1 --2- 172.21.15.148:0/3133947489 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f48480ad000 0x7f48481368b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.710+0000 7f4847fff640 1 -- 172.21.15.148:0/3133947489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48480acc20 msgr2=0x7f48481da7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.710+0000 7f4847fff640 1 --2- 172.21.15.148:0/3133947489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48480acc20 0x7f48481da7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.710+0000 7f4847fff640 10 monclient: _finish_hunting 0 2023-10-30T12:39:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.710+0000 7f4847fff640 1 monclient: found mon.noname-a 2023-10-30T12:39:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.710+0000 7f4847fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.710+0000 7f4847fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:39:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.710+0000 7f4847fff640 1 -- 172.21.15.148:0/3133947489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4848136df0 con 0x7f48481dad10 2023-10-30T12:39:52.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.724+0000 7f4847fff640 1 --2- 172.21.15.148:0/3133947489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48481dad10 0x7f4848136370 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7f483400c390 tx=0x7f483400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:52.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.724+0000 7f48457fa640 10 client.?.objecter ms_handle_connect 0x7f48481dad10 2023-10-30T12:39:52.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.724+0000 7f48457fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:52.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.755+0000 7f48457fa640 1 -- 172.21.15.148:0/3133947489 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48340035f0 con 0x7f48481dad10 2023-10-30T12:39:52.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.755+0000 7f48457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:52.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.755+0000 7f48457fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:39:52.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.755+0000 7f48457fa640 10 monclient: dump: 2023-10-30T12:39:52.758 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:52.759 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.759 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.759 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.759 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:52.759 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.759 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:52.760 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:52.760 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:52.760 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.755+0000 7f48457fa640 10 monclient: _finish_auth 0 2023-10-30T12:39:52.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.755+0000 7f48457fa640 10 monclient: _check_auth_tickets 2023-10-30T12:39:52.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f48457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:52.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f48457fa640 1 -- 172.21.15.148:0/3133947489 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483400e040 con 0x7f48481dad10 2023-10-30T12:39:52.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 5 monclient: authenticate success, global_id 16305 2023-10-30T12:39:52.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f48457fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:52.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f48457fa640 1 -- 172.21.15.148:0/3133947489 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48340039a0 con 0x7f48481dad10 2023-10-30T12:39:52.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f48457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:52.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f48457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:39:52.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f48457fa640 10 monclient: dump: 2023-10-30T12:39:52.762 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:52.762 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:52.762 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.762 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:52.763 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:52.763 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:52.763 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:52.763 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:52.763 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:52.763 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:52.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 10 monclient: _renew_subs 2023-10-30T12:39:52.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:39:52.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48480ab260 con 0x7f48481dad10 2023-10-30T12:39:52.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f4846ffd640 4 set_mon_vals no callback set 2023-10-30T12:39:52.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 10 client.16305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:52.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 10 monclient: _renew_subs 2023-10-30T12:39:52.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:39:52.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48480ac080 con 0x7f48481dad10 2023-10-30T12:39:52.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 1 librados: init done 2023-10-30T12:39:52.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 10 client.16305.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:52.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 15 client.16305.objecter _session_command_op_assign -1 1 2023-10-30T12:39:52.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 20 client.16305.objecter _get_session osd=-1 returning homeless 2023-10-30T12:39:52.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 10 client.16305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:52.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:39:52.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.756+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f48480a97e0 con 0x7f48481dad10 2023-10-30T12:39:52.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.792+0000 7f48457fa640 1 -- 172.21.15.148:0/3133947489 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f4834003b10 con 0x7f48481dad10 2023-10-30T12:39:52.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.793+0000 7f48457fa640 1 --2- 172.21.15.148:0/3133947489 >> v2:172.21.15.148:6800/3205090179 conn(0x7f482c06c7f0 0x7f482c06ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.793+0000 7f484cbed640 1 --2- 172.21.15.148:0/3133947489 >> v2:172.21.15.148:6800/3205090179 conn(0x7f482c06c7f0 0x7f482c06ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.793+0000 7f484cbed640 10 monclient: get_auth_request con 0x7f482c06c7f0 auth_method 0 2023-10-30T12:39:52.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.793+0000 7f484cbed640 1 --2- 172.21.15.148:0/3133947489 >> v2:172.21.15.148:6800/3205090179 conn(0x7f482c06c7f0 0x7f482c06ecb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f483c001050 tx=0x7f483c001d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:52.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.793+0000 7f48457fa640 10 client.16305.objecter ms_handle_connect 0x7f482c06c7f0 2023-10-30T12:39:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 1 -- 172.21.15.148:0/3133947489 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(900..900 src has 739..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f4834057b60 con 0x7f48481dad10 2023-10-30T12:39:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 10 client.16305.objecter ms_dispatch 0x7f4848000b90 osd_map(900..900 src has 739..900) v4 2023-10-30T12:39:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 3 client.16305.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 3 client.16305.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 10 client.16305.objecter checking command 1 2023-10-30T12:39:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 1 --2- 172.21.15.148:0/3133947489 >> v2:172.21.15.150:6800/2604225934 conn(0x7f482c073b10 0x7f482c075fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 20 client.16305.objecter _get_session s=0x7f48340039a0 osd=6 3 2023-10-30T12:39:52.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 20 client.16305.objecter put_session s=0x7f48340039a0 osd=6 3 2023-10-30T12:39:52.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 15 client.16305.objecter _session_command_op_remove -1 1 2023-10-30T12:39:52.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 20 client.16305.objecter _get_session s=0x7f48340039a0 osd=6 3 2023-10-30T12:39:52.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 20 client.16305.objecter get_session s=0x7f48340039a0 osd=6 3 2023-10-30T12:39:52.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 15 client.16305.objecter _session_command_op_assign 6 1 2023-10-30T12:39:52.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 20 client.16305.objecter put_session s=0x7f48340039a0 osd=6 4 2023-10-30T12:39:52.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 10 client.16305.objecter _send_command 1 2023-10-30T12:39:52.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 1 -- 172.21.15.148:0/3133947489 --> v2:172.21.15.150:6800/2604225934 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48340035f0 con 0x7f482c073b10 2023-10-30T12:39:52.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 20 client.16305.objecter dump_active .. 0 homeless 2023-10-30T12:39:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 1 -- 172.21.15.148:0/3133947489 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48340010c0 con 0x7f48481dad10 2023-10-30T12:39:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f48457fa640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:39:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.799+0000 7f484d3ee640 1 --2- 172.21.15.148:0/3133947489 >> v2:172.21.15.150:6800/2604225934 conn(0x7f482c073b10 0x7f482c075fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.800+0000 7f484d3ee640 10 monclient: get_auth_request con 0x7f482c073b10 auth_method 0 2023-10-30T12:39:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.803+0000 7f48457fa640 10 client.16305.objecter ms_handle_connect 0x7f482c073b10 2023-10-30T12:39:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.803+0000 7f484d3ee640 1 --2- 172.21.15.148:0/3133947489 >> v2:172.21.15.150:6800/2604225934 conn(0x7f482c073b10 0x7f482c075fb0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:52.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.828+0000 7f48457fa640 1 -- 172.21.15.148:0/3133947489 <== osd.6 v2:172.21.15.150:6800/2604225934 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f48340035f0 con 0x7f482c073b10 2023-10-30T12:39:52.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.828+0000 7f48457fa640 10 client.16305.objecter ms_dispatch 0x7f4848000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:39:52.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.828+0000 7f48457fa640 10 client.16305.objecter _finish_command 1 = system:0 2023-10-30T12:39:52.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.828+0000 7f48457fa640 20 client.16305.objecter put_session s=0x7f48340039a0 osd=6 4 2023-10-30T12:39:52.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.828+0000 7f48457fa640 15 client.16305.objecter _session_command_op_remove 6 1 2023-10-30T12:39:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 10 client.16305.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:39:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 15 client.16305.objecter _session_command_op_assign -1 2 2023-10-30T12:39:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 20 client.16305.objecter _get_session s=0x7f48340039a0 osd=6 3 2023-10-30T12:39:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 20 client.16305.objecter put_session s=0x7f48340039a0 osd=6 3 2023-10-30T12:39:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 20 client.16305.objecter _get_session s=0x7f48340039a0 osd=6 3 2023-10-30T12:39:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 15 client.16305.objecter _session_command_op_remove -1 2 2023-10-30T12:39:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 20 client.16305.objecter get_session s=0x7f48340039a0 osd=6 3 2023-10-30T12:39:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 15 client.16305.objecter _session_command_op_assign 6 2 2023-10-30T12:39:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 20 client.16305.objecter put_session s=0x7f48340039a0 osd=6 4 2023-10-30T12:39:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 10 client.16305.objecter _send_command 2 2023-10-30T12:39:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.842+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 --> v2:172.21.15.150:6800/2604225934 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f48480b7ff0 con 0x7f482c073b10 2023-10-30T12:39:52.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.856+0000 7f48457fa640 1 -- 172.21.15.148:0/3133947489 <== osd.6 v2:172.21.15.150:6800/2604225934 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4840034440 con 0x7f482c073b10 2023-10-30T12:39:52.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.856+0000 7f48457fa640 10 client.16305.objecter ms_dispatch 0x7f4848000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:39:52.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.856+0000 7f48457fa640 10 client.16305.objecter _finish_command 2 = system:0 2023-10-30T12:39:52.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.856+0000 7f48457fa640 20 client.16305.objecter put_session s=0x7f48340039a0 osd=6 4 2023-10-30T12:39:52.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.856+0000 7f48457fa640 15 client.16305.objecter _session_command_op_remove 6 2 2023-10-30T12:39:52.859 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T12:39:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:39:52.859 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T12:39:52.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 10 librados: watch_flush enter 2023-10-30T12:39:52.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 10 librados: watch_flush exit 2023-10-30T12:39:52.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 10 client.16305.objecter close_session for osd.6 2023-10-30T12:39:52.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 >> v2:172.21.15.150:6800/2604225934 conn(0x7f482c073b10 msgr2=0x7f482c075fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 --2- 172.21.15.148:0/3133947489 >> v2:172.21.15.150:6800/2604225934 conn(0x7f482c073b10 0x7f482c075fb0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 20 client.16305.objecter put_session s=0x7f48340039a0 osd=6 1 2023-10-30T12:39:52.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 20 client.16305.objecter shutdown clearing up homeless session... 2023-10-30T12:39:52.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 10 client.16305.objecter successfully canceled tick 2023-10-30T12:39:52.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 >> v2:172.21.15.148:6800/3205090179 conn(0x7f482c06c7f0 msgr2=0x7f482c06ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 --2- 172.21.15.148:0/3133947489 >> v2:172.21.15.148:6800/3205090179 conn(0x7f482c06c7f0 0x7f482c06ecb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f483c001050 tx=0x7f483c001d50 comp rx=0 tx=0).stop 2023-10-30T12:39:52.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 10 monclient: shutdown 2023-10-30T12:39:52.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:52.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48481dad10 msgr2=0x7f4848136370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 --2- 172.21.15.148:0/3133947489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48481dad10 0x7f4848136370 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7f483400c390 tx=0x7f483400c850 comp rx=0 tx=0).stop 2023-10-30T12:39:52.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484d3ee640 1 -- 172.21.15.148:0/3133947489 reap_dead start 2023-10-30T12:39:52.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 shutdown_connections 2023-10-30T12:39:52.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 >> 172.21.15.148:0/3133947489 conn(0x7f484806c660 msgr2=0x7f48480738f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:52.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 shutdown_connections 2023-10-30T12:39:52.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 -- 172.21.15.148:0/3133947489 wait complete. 2023-10-30T12:39:52.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:52.857+0000 7f484ee51640 1 librados: shutdown 2023-10-30T12:39:52.872 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-30T12:39:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.019+0000 7f72fc694640 10 monclient: get_monmap_and_config 2023-10-30T12:39:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.019+0000 7f72fc694640 10 monclient: build_initial_monmap 2023-10-30T12:39:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.019+0000 7f72fc694640 10 monclient: monmap: 2023-10-30T12:39:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:53.020543+0000 2023-10-30T12:39:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:53.020543+0000 2023-10-30T12:39:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:53.024 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 1 Processor -- start 2023-10-30T12:39:53.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 1 -- start start 2023-10-30T12:39:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient: init 2023-10-30T12:39:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-10-30T12:39:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 0x7f72f40a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient(hunting): picked mon.noname-b con 0x7f72f40b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:53.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f72f4141940 0x7f72f4141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient(hunting): picked mon.noname-a con 0x7f72f4141940 addr v2:172.21.15.148:3300/0 2023-10-30T12:39:53.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72f40b1c10 0x7f72f40b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient(hunting): picked mon.noname-c con 0x7f72f40b1c10 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:53.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f40670a0 con 0x7f72f4141940 2023-10-30T12:39:53.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f40b7ff0 con 0x7f72f40b1c10 2023-10-30T12:39:53.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f40a9350 con 0x7f72f40b7c10 2023-10-30T12:39:53.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:53.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.020+0000 7f72fc694640 10 monclient(hunting): authenticate will time out at 2885.225773s 2023-10-30T12:39:53.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.022+0000 7f72f9c2f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f72f4141940 0x7f72f4141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.023+0000 7f72f9c2f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f72f4141940 0x7f72f4141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52352/0 (socket says 172.21.15.148:52352) 2023-10-30T12:39:53.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.023+0000 7f72f9c2f640 1 -- 172.21.15.148:0/4276433116 learned_addr learned my addr 172.21.15.148:0/4276433116 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:53.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.023+0000 7f72f9c2f640 10 monclient(hunting): get_auth_request con 0x7f72f4141940 auth_method 0 2023-10-30T12:39:53.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.023+0000 7f72f9c2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:53.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.023+0000 7f72f9c2f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:53.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.023+0000 7f72f9c2f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:53.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.023+0000 7f72f9c2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:53.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.023+0000 7f72f9c2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:53.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.023+0000 7f72f9c2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:53.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.024+0000 7f72fa430640 1 --2- 172.21.15.148:0/4276433116 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 0x7f72f40a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.026+0000 7f72f9c2f640 10 monclient(hunting): handle_auth_done global_id 16308 payload 274 2023-10-30T12:39:53.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.026+0000 7f72f9c2f640 1 -- 172.21.15.148:0/4276433116 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 msgr2=0x7f72f40a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:53.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.026+0000 7f72f9c2f640 1 --2- 172.21.15.148:0/4276433116 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 0x7f72f40a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.026+0000 7f72f9c2f640 1 -- 172.21.15.148:0/4276433116 >> v2:172.21.15.150:3300/0 conn(0x7f72f40b1c10 msgr2=0x7f72f40b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:53.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.026+0000 7f72f9c2f640 1 --2- 172.21.15.148:0/4276433116 >> v2:172.21.15.150:3300/0 conn(0x7f72f40b1c10 0x7f72f40b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.026+0000 7f72f9c2f640 10 monclient: _finish_hunting 0 2023-10-30T12:39:53.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.026+0000 7f72f9c2f640 1 monclient: found mon.noname-a 2023-10-30T12:39:53.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.026+0000 7f72f9c2f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:53.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.026+0000 7f72f9c2f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:39:53.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.026+0000 7f72f9c2f640 1 -- 172.21.15.148:0/4276433116 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f40aa6a0 con 0x7f72f4141940 2023-10-30T12:39:53.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.029+0000 7f72f9c2f640 1 --2- 172.21.15.148:0/4276433116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f4141940 0x7f72f4141d60 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 crypto rx=0x7f72f40b3c70 tx=0x7f72e402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c626a2d90df6cd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 1 -- 172.21.15.148:0/4276433116 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e40317e0 con 0x7f72f4141940 2023-10-30T12:39:53.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:39:53.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 10 monclient: dump: 2023-10-30T12:39:53.045 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.046 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.046 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.046 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.046 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.046 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.047 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.047 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.047 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.047 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 10 monclient: _finish_auth 0 2023-10-30T12:39:53.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 10 monclient: _check_auth_tickets 2023-10-30T12:39:53.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:53.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 1 -- 172.21.15.148:0/4276433116 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e4031950 con 0x7f72f4141940 2023-10-30T12:39:53.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72fc694640 5 monclient: authenticate success, global_id 16308 2023-10-30T12:39:53.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72fc694640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:39:53.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:53.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72fc694640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:53.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 1 -- 172.21.15.148:0/4276433116 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4031cf0 con 0x7f72f4141940 2023-10-30T12:39:53.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72fc694640 4 set_mon_vals no callback set 2023-10-30T12:39:53.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:39:53.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72f942e640 10 monclient: dump: 2023-10-30T12:39:53.050 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.050 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.050 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.050 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.050 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.051 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.051 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.051 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.051 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.051 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72fc694640 10 monclient: shutdown 2023-10-30T12:39:53.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72fc694640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:53.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72fc694640 1 -- 172.21.15.148:0/4276433116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f4141940 msgr2=0x7f72f4141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.043+0000 7f72fc694640 1 --2- 172.21.15.148:0/4276433116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f4141940 0x7f72f4141d60 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 crypto rx=0x7f72f40b3c70 tx=0x7f72e402b880 comp rx=0 tx=0).stop 2023-10-30T12:39:53.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.044+0000 7f72fc694640 1 -- 172.21.15.148:0/4276433116 shutdown_connections 2023-10-30T12:39:53.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.044+0000 7f72fc694640 1 --2- 172.21.15.148:0/4276433116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f4141940 0x7f72f4141d60 unknown :-1 s=CLOSED pgs=3234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.044+0000 7f72fc694640 1 --2- 172.21.15.148:0/4276433116 >> v2:172.21.15.150:3300/0 conn(0x7f72f40b1c10 0x7f72f40b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.044+0000 7f72fc694640 1 --2- 172.21.15.148:0/4276433116 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 0x7f72f40a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.044+0000 7f72fc694640 1 -- 172.21.15.148:0/4276433116 >> 172.21.15.148:0/4276433116 conn(0x7f72f406c660 msgr2=0x7f72f40b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:53.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.044+0000 7f72fc694640 1 -- 172.21.15.148:0/4276433116 shutdown_connections 2023-10-30T12:39:53.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.044+0000 7f72fc694640 1 -- 172.21.15.148:0/4276433116 wait complete. 2023-10-30T12:39:53.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.044+0000 7f72fc694640 10 monclient: build_initial_monmap 2023-10-30T12:39:53.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.044+0000 7f72fc694640 10 monclient: monmap: 2023-10-30T12:39:53.056 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:53.056 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:53.057 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:53.057 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:53.057 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:53.057 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:53.057 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:53.057 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:53.058 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:53.058 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 librados: starting msgr at 2023-10-30T12:39:53.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 librados: starting objecter 2023-10-30T12:39:53.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 Processor -- start 2023-10-30T12:39:53.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 -- start start 2023-10-30T12:39:53.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 librados: setting wanted keys 2023-10-30T12:39:53.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 librados: calling monclient init 2023-10-30T12:39:53.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient: init 2023-10-30T12:39:53.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:53.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:39:53.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f40b1c10 0x7f72f40b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient(hunting): picked mon.noname-a con 0x7f72f40b1c10 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:53.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 0x7f72f411f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient(hunting): picked mon.noname-c con 0x7f72f40b7c10 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:53.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f4141940 0x7f72f41d6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient(hunting): picked mon.noname-b con 0x7f72f4141940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:53.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f40b29d0 con 0x7f72f40b1c10 2023-10-30T12:39:53.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f4120e00 con 0x7f72f4141940 2023-10-30T12:39:53.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f41d69d0 con 0x7f72f40b7c10 2023-10-30T12:39:53.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:53.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.045+0000 7f72fc694640 10 monclient(hunting): authenticate will time out at 2885.250693s 2023-10-30T12:39:53.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.047+0000 7f72f9c2f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 0x7f72f411f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.048+0000 7f72f9c2f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 0x7f72f411f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:51626/0 (socket says 172.21.15.148:51626) 2023-10-30T12:39:53.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.048+0000 7f72f9c2f640 1 -- 172.21.15.148:0/893025341 learned_addr learned my addr 172.21.15.148:0/893025341 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:53.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.048+0000 7f72f9c2f640 10 monclient(hunting): get_auth_request con 0x7f72f40b7c10 auth_method 0 2023-10-30T12:39:53.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.048+0000 7f72f9c2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:53.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.048+0000 7f72f9c2f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:53.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.048+0000 7f72f9c2f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:53.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.050+0000 7f72f9c2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:53.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.050+0000 7f72f9c2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:53.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.050+0000 7f72f9c2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:53.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.054+0000 7f72fac31640 1 --2- 172.21.15.148:0/893025341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f4141940 0x7f72f41d6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.054+0000 7f72f9c2f640 10 monclient(hunting): handle_auth_done global_id 17759 payload 824 2023-10-30T12:39:53.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.054+0000 7f72f9c2f640 1 -- 172.21.15.148:0/893025341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f4141940 msgr2=0x7f72f41d6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.054+0000 7f72f9c2f640 1 --2- 172.21.15.148:0/893025341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f4141940 0x7f72f41d6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.055+0000 7f72f9c2f640 1 -- 172.21.15.148:0/893025341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f40b1c10 msgr2=0x7f72f40b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.055+0000 7f72f9c2f640 1 --2- 172.21.15.148:0/893025341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f40b1c10 0x7f72f40b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.055+0000 7f72f9c2f640 10 monclient: _finish_hunting 0 2023-10-30T12:39:53.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.055+0000 7f72f9c2f640 1 monclient: found mon.noname-c 2023-10-30T12:39:53.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.055+0000 7f72f9c2f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:53.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.055+0000 7f72f9c2f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:39:53.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.055+0000 7f72f9c2f640 1 -- 172.21.15.148:0/893025341 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f40adfb0 con 0x7f72f40b7c10 2023-10-30T12:39:53.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.058+0000 7f72f9c2f640 1 --2- 172.21.15.148:0/893025341 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 0x7f72f411f390 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f72e4000c00 tx=0x7f72e40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.058+0000 7f72eaffd640 10 client.?.objecter ms_handle_connect 0x7f72f40b7c10 2023-10-30T12:39:53.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.058+0000 7f72eaffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:53.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 1 -- 172.21.15.148:0/893025341 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4032a00 con 0x7f72f40b7c10 2023-10-30T12:39:53.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:53.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 10 monclient: dump: 2023-10-30T12:39:53.072 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.072 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.073 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.073 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.073 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.073 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.073 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.073 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.074 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.074 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 10 monclient: _finish_auth 0 2023-10-30T12:39:53.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:39:53.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:53.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 1 -- 172.21.15.148:0/893025341 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e403f030 con 0x7f72f40b7c10 2023-10-30T12:39:53.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:53.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 1 -- 172.21.15.148:0/893025341 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4032db0 con 0x7f72f40b7c10 2023-10-30T12:39:53.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72fc694640 5 monclient: authenticate success, global_id 17759 2023-10-30T12:39:53.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72fc694640 10 monclient: _renew_subs 2023-10-30T12:39:53.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72fc694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:53.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72fc694640 1 -- 172.21.15.148:0/893025341 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f40afee0 con 0x7f72f40b7c10 2023-10-30T12:39:53.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:53.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72eaffd640 10 monclient: dump: 2023-10-30T12:39:53.076 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.077 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.077 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.077 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.077 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.077 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.078 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.078 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.078 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.078 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.069+0000 7f72f8c2d640 4 set_mon_vals no callback set 2023-10-30T12:39:53.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.070+0000 7f72fc694640 10 client.17759.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:53.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.070+0000 7f72fc694640 10 monclient: _renew_subs 2023-10-30T12:39:53.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.070+0000 7f72fc694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:53.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.070+0000 7f72fc694640 1 -- 172.21.15.148:0/893025341 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f41d7360 con 0x7f72f40b7c10 2023-10-30T12:39:53.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.070+0000 7f72fc694640 1 librados: init done 2023-10-30T12:39:53.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.070+0000 7f72fc694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:53.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.070+0000 7f72fc694640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:53.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.070+0000 7f72fc694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:53.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.070+0000 7f72fc694640 1 -- 172.21.15.148:0/893025341 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f41d6b20 con 0x7f72f40b7c10 2023-10-30T12:39:53.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.344+0000 7f72eaffd640 1 -- 172.21.15.148:0/893025341 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f72e404d3f0 con 0x7f72f40b7c10 2023-10-30T12:39:53.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.345+0000 7f72eaffd640 1 --2- 172.21.15.148:0/893025341 >> v2:172.21.15.148:6800/3205090179 conn(0x7f72d806c7c0 0x7f72d806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.345+0000 7f72fa430640 1 --2- 172.21.15.148:0/893025341 >> v2:172.21.15.148:6800/3205090179 conn(0x7f72d806c7c0 0x7f72d806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.345+0000 7f72fa430640 10 monclient: get_auth_request con 0x7f72d806c7c0 auth_method 0 2023-10-30T12:39:53.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.345+0000 7f72eaffd640 10 client.17759.objecter ms_handle_connect 0x7f72d806c7c0 2023-10-30T12:39:53.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.345+0000 7f72fa430640 1 --2- 172.21.15.148:0/893025341 >> v2:172.21.15.148:6800/3205090179 conn(0x7f72d806c7c0 0x7f72d806ec80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f72f0006fd0 tx=0x7f72f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.358+0000 7f72eaffd640 1 -- 172.21.15.148:0/893025341 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(900..900 src has 739..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f72e4031900 con 0x7f72f40b7c10 2023-10-30T12:39:53.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.358+0000 7f72eaffd640 10 client.17759.objecter ms_dispatch 0x7f72f4000b90 osd_map(900..900 src has 739..900) v4 2023-10-30T12:39:53.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.358+0000 7f72eaffd640 3 client.17759.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:53.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.358+0000 7f72eaffd640 3 client.17759.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:53.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.358+0000 7f72eaffd640 20 client.17759.objecter dump_active .. 0 homeless 2023-10-30T12:39:53.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.358+0000 7f72eaffd640 1 -- 172.21.15.148:0/893025341 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f72e4037070 con 0x7f72f40b7c10 2023-10-30T12:39:53.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.358+0000 7f72eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:53.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.358+0000 7f72eaffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:53.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.544+0000 7f72fc694640 1 -- 172.21.15.148:0/893025341 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f72f40a8fe0 con 0x7f72d806c7c0 2023-10-30T12:39:53.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.544+0000 7f72eaffd640 1 -- 172.21.15.148:0/893025341 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f72f000ae20 con 0x7f72d806c7c0 2023-10-30T12:39:53.547 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 6 to deep-scrub 2023-10-30T12:39:53.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 10 librados: watch_flush enter 2023-10-30T12:39:53.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 10 librados: watch_flush exit 2023-10-30T12:39:53.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 20 client.17759.objecter shutdown clearing up homeless session... 2023-10-30T12:39:53.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 10 client.17759.objecter successfully canceled tick 2023-10-30T12:39:53.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 -- 172.21.15.148:0/893025341 >> v2:172.21.15.148:6800/3205090179 conn(0x7f72d806c7c0 msgr2=0x7f72d806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 --2- 172.21.15.148:0/893025341 >> v2:172.21.15.148:6800/3205090179 conn(0x7f72d806c7c0 0x7f72d806ec80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f72f0006fd0 tx=0x7f72f0008040 comp rx=0 tx=0).stop 2023-10-30T12:39:53.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 10 monclient: shutdown 2023-10-30T12:39:53.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:53.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 -- 172.21.15.148:0/893025341 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 msgr2=0x7f72f411f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 --2- 172.21.15.148:0/893025341 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 0x7f72f411f390 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f72e4000c00 tx=0x7f72e40046c0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 -- 172.21.15.148:0/893025341 shutdown_connections 2023-10-30T12:39:53.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 --2- 172.21.15.148:0/893025341 >> v2:172.21.15.148:6800/3205090179 conn(0x7f72d806c7c0 0x7f72d806ec80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 --2- 172.21.15.148:0/893025341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f4141940 0x7f72f41d6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 --2- 172.21.15.148:0/893025341 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f72f40b7c10 0x7f72f411f390 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 --2- 172.21.15.148:0/893025341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f40b1c10 0x7f72f40b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 -- 172.21.15.148:0/893025341 >> 172.21.15.148:0/893025341 conn(0x7f72f406c660 msgr2=0x7f72f413c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:53.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 -- 172.21.15.148:0/893025341 shutdown_connections 2023-10-30T12:39:53.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 -- 172.21.15.148:0/893025341 wait complete. 2023-10-30T12:39:53.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.546+0000 7f72fc694640 1 librados: shutdown 2023-10-30T12:39:53.560 INFO:tasks.ceph:Scrubbing osd.7 2023-10-30T12:39:53.561 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:39:53.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.662+0000 7feabddb0640 10 monclient: get_monmap_and_config 2023-10-30T12:39:53.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.662+0000 7feabddb0640 10 monclient: build_initial_monmap 2023-10-30T12:39:53.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.662+0000 7feabddb0640 10 monclient: monmap: 2023-10-30T12:39:53.664 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:53.665 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.665 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:53.663498+0000 2023-10-30T12:39:53.665 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:53.663498+0000 2023-10-30T12:39:53.665 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:53.665 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.666 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:53.666 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:53.666 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:53.666 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:53.666 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:53.667 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 1 Processor -- start 2023-10-30T12:39:53.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 1 -- start start 2023-10-30T12:39:53.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 10 monclient: init 2023-10-30T12:39:53.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:53.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-10-30T12:39:53.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feab8065d20 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:53.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feab8071280 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:53.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7feab813ce40 0x7feab8143400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 10 monclient(hunting): picked mon.noname-a con 0x7feab813ce40 addr v2:172.21.15.148:3300/0 2023-10-30T12:39:53.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab8070460 con 0x7feab8065d20 2023-10-30T12:39:53.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab8143940 con 0x7feab8071280 2023-10-30T12:39:53.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab8067110 con 0x7feab813ce40 2023-10-30T12:39:53.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:53.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.663+0000 7feabddb0640 10 monclient(hunting): authenticate will time out at 2885.868801s 2023-10-30T12:39:53.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.664+0000 7feab7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7feab813ce40 0x7feab8143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.664+0000 7feab6ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7feab8071280 0x7feab8142cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:45366/0 (socket says 172.21.15.148:45366) 2023-10-30T12:39:53.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.664+0000 7feab6ffd640 1 -- 172.21.15.148:0/3096339907 learned_addr learned my addr 172.21.15.148:0/3096339907 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:39:53.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.664+0000 7feab7fff640 10 monclient(hunting): get_auth_request con 0x7feab813ce40 auth_method 0 2023-10-30T12:39:53.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.664+0000 7feab7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:53.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.664+0000 7feab7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:53.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.664+0000 7feab7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:53.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.665+0000 7feab7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:53.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.665+0000 7feab7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:53.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.665+0000 7feab7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:53.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.667+0000 7feab7fff640 10 monclient(hunting): handle_auth_done global_id 16311 payload 274 2023-10-30T12:39:53.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.667+0000 7feab7fff640 1 -- 172.21.15.148:0/3096339907 >> v1:172.21.15.150:6789/0 conn(0x7feab8071280 legacy=0x7feab8142cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.667+0000 7feab7fff640 1 -- 172.21.15.148:0/3096339907 >> v1:172.21.15.148:6789/0 conn(0x7feab8065d20 legacy=0x7feab813c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:53.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.667+0000 7feab7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:39:53.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.667+0000 7feab7fff640 1 monclient: found mon.noname-a 2023-10-30T12:39:53.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.667+0000 7feab67fc640 1 -- 172.21.15.148:0/3096339907 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102506732 0 0) 0x7feaa8001110 con 0x7feab8071280 2023-10-30T12:39:53.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.667+0000 7feab7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:53.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.667+0000 7feab7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:39:53.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.667+0000 7feab7fff640 1 -- 172.21.15.148:0/3096339907 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80a8570 con 0x7feab813ce40 2023-10-30T12:39:53.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.667+0000 7feab67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:39:53.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.669+0000 7feab7fff640 1 --2- 172.21.15.148:0/3096339907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feab813ce40 0x7feab8143400 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7feab0009d90 tx=0x7feab002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b8764597d91d10 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 1 -- 172.21.15.148:0/3096339907 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab002cd00 con 0x7feab813ce40 2023-10-30T12:39:53.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:39:53.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 10 monclient: dump: 2023-10-30T12:39:53.680 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.680 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.680 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.680 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.681 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 10 monclient: _finish_auth 0 2023-10-30T12:39:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:39:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feabddb0640 5 monclient: authenticate success, global_id 16311 2023-10-30T12:39:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 1 -- 172.21.15.148:0/3096339907 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab002ce70 con 0x7feab813ce40 2023-10-30T12:39:53.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:53.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feabddb0640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:53.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feabddb0640 4 set_mon_vals no callback set 2023-10-30T12:39:53.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 1 -- 172.21.15.148:0/3096339907 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0034660 con 0x7feab813ce40 2023-10-30T12:39:53.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:39:53.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.673+0000 7feab67fc640 10 monclient: dump: 2023-10-30T12:39:53.684 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.684 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.684 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.684 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.685 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.685 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.685 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.685 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.685 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.686 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.674+0000 7feabddb0640 10 monclient: shutdown 2023-10-30T12:39:53.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.674+0000 7feabddb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:53.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.674+0000 7feabddb0640 1 -- 172.21.15.148:0/3096339907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feab813ce40 msgr2=0x7feab8143400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.674+0000 7feabddb0640 1 --2- 172.21.15.148:0/3096339907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feab813ce40 0x7feab8143400 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7feab0009d90 tx=0x7feab002bf50 comp rx=0 tx=0).stop 2023-10-30T12:39:53.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.674+0000 7feabddb0640 1 -- 172.21.15.148:0/3096339907 shutdown_connections 2023-10-30T12:39:53.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.674+0000 7feabddb0640 1 --2- 172.21.15.148:0/3096339907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feab813ce40 0x7feab8143400 unknown :-1 s=CLOSED pgs=3235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.674+0000 7feabddb0640 1 -- 172.21.15.148:0/3096339907 >> 172.21.15.148:0/3096339907 conn(0x7feab806a1e0 msgr2=0x7feab806a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:53.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.674+0000 7feabddb0640 1 -- 172.21.15.148:0/3096339907 shutdown_connections 2023-10-30T12:39:53.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.674+0000 7feabddb0640 1 -- 172.21.15.148:0/3096339907 wait complete. 2023-10-30T12:39:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.675+0000 7feabddb0640 10 monclient: build_initial_monmap 2023-10-30T12:39:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.675+0000 7feabddb0640 10 monclient: monmap: 2023-10-30T12:39:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:53.690 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.675+0000 7feabddb0640 1 librados: starting msgr at 2023-10-30T12:39:53.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.675+0000 7feabddb0640 1 librados: starting objecter 2023-10-30T12:39:53.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.675+0000 7feabddb0640 1 Processor -- start 2023-10-30T12:39:53.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.675+0000 7feabddb0640 1 -- start start 2023-10-30T12:39:53.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.675+0000 7feabddb0640 1 librados: setting wanted keys 2023-10-30T12:39:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.675+0000 7feabddb0640 1 librados: calling monclient init 2023-10-30T12:39:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.675+0000 7feabddb0640 10 monclient: init 2023-10-30T12:39:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:39:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feab8065d20 0x7feab81d8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 10 monclient(hunting): picked mon.noname-a con 0x7feab8065d20 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:53.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feab813ce40 0x7feab8138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 10 monclient(hunting): picked mon.noname-b con 0x7feab813ce40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:53.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7feab81d85d0 0x7feab81d4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 10 monclient(hunting): picked mon.noname-c con 0x7feab81d85d0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:53.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab80aff50 con 0x7feab8065d20 2023-10-30T12:39:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab81d7b70 con 0x7feab813ce40 2023-10-30T12:39:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab8140060 con 0x7feab81d85d0 2023-10-30T12:39:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feabddb0640 10 monclient(hunting): authenticate will time out at 2885.881067s 2023-10-30T12:39:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feab77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feab8065d20 0x7feab81d8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feab7fff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7feab81d85d0 0x7feab81d4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feab77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feab8065d20 0x7feab81d8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52372/0 (socket says 172.21.15.148:52372) 2023-10-30T12:39:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feab77fe640 1 -- 172.21.15.148:0/3168826244 learned_addr learned my addr 172.21.15.148:0/3168826244 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:53.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feab77fe640 10 monclient(hunting): get_auth_request con 0x7feab8065d20 auth_method 0 2023-10-30T12:39:53.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feab77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:53.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feab77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:53.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feab77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:53.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.676+0000 7feab6ffd640 1 --2- 172.21.15.148:0/3168826244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feab813ce40 0x7feab8138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab7fff640 10 monclient(hunting): get_auth_request con 0x7feab81d85d0 auth_method 0 2023-10-30T12:39:53.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:53.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:53.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:53.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab6ffd640 10 monclient(hunting): get_auth_request con 0x7feab813ce40 auth_method 0 2023-10-30T12:39:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:53.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:53.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:53.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:53.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:53.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:53.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.677+0000 7feab6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:53.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.680+0000 7feab77fe640 10 monclient(hunting): handle_auth_done global_id 16314 payload 824 2023-10-30T12:39:53.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.680+0000 7feab77fe640 1 -- 172.21.15.148:0/3168826244 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7feab81d85d0 msgr2=0x7feab81d4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.680+0000 7feab77fe640 1 --2- 172.21.15.148:0/3168826244 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7feab81d85d0 0x7feab81d4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.680+0000 7feab77fe640 1 -- 172.21.15.148:0/3168826244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feab813ce40 msgr2=0x7feab8138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.680+0000 7feab77fe640 1 --2- 172.21.15.148:0/3168826244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feab813ce40 0x7feab8138dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.680+0000 7feab77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:39:53.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.680+0000 7feab77fe640 1 monclient: found mon.noname-a 2023-10-30T12:39:53.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.680+0000 7feab77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:53.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.680+0000 7feab77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:39:53.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.680+0000 7feab77fe640 1 -- 172.21.15.148:0/3168826244 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab81387e0 con 0x7feab8065d20 2023-10-30T12:39:53.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.682+0000 7feab77fe640 1 --2- 172.21.15.148:0/3168826244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feab8065d20 0x7feab81d8090 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 crypto rx=0x7feaac00c410 tx=0x7feaac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.682+0000 7fea9ffff640 10 client.?.objecter ms_handle_connect 0x7feab8065d20 2023-10-30T12:39:53.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.682+0000 7fea9ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 1 -- 172.21.15.148:0/3168826244 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac0035f0 con 0x7feab8065d20 2023-10-30T12:39:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:39:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 10 monclient: dump: 2023-10-30T12:39:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.705 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 10 monclient: _finish_auth 0 2023-10-30T12:39:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:39:53.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:53.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 1 -- 172.21.15.148:0/3168826244 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac00e040 con 0x7feab8065d20 2023-10-30T12:39:53.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:53.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 1 -- 172.21.15.148:0/3168826244 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac0039e0 con 0x7feab8065d20 2023-10-30T12:39:53.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7feabddb0640 5 monclient: authenticate success, global_id 16314 2023-10-30T12:39:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7feabddb0640 10 monclient: _renew_subs 2023-10-30T12:39:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7feabddb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:39:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab8139300 con 0x7feab8065d20 2023-10-30T12:39:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:39:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7fea9ffff640 10 monclient: dump: 2023-10-30T12:39:53.708 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.708 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.708 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.708 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.708 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.709 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.709 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.709 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.709 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.709 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7feab5ffb640 4 set_mon_vals no callback set 2023-10-30T12:39:53.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7feabddb0640 10 client.16314.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:53.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7feabddb0640 10 monclient: _renew_subs 2023-10-30T12:39:53.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7feabddb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:39:53.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab81d4960 con 0x7feab8065d20 2023-10-30T12:39:53.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.686+0000 7feabddb0640 1 librados: init done 2023-10-30T12:39:53.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.687+0000 7feabddb0640 10 client.16314.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:53.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.687+0000 7feabddb0640 15 client.16314.objecter _session_command_op_assign -1 1 2023-10-30T12:39:53.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.687+0000 7feabddb0640 20 client.16314.objecter _get_session osd=-1 returning homeless 2023-10-30T12:39:53.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.687+0000 7feabddb0640 10 client.16314.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:53.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.687+0000 7feabddb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:39:53.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.687+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feab80a8850 con 0x7feab8065d20 2023-10-30T12:39:53.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.706+0000 7fea9ffff640 1 -- 172.21.15.148:0/3168826244 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7feaac003b50 con 0x7feab8065d20 2023-10-30T12:39:53.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.706+0000 7fea9ffff640 1 --2- 172.21.15.148:0/3168826244 >> v2:172.21.15.148:6800/3205090179 conn(0x7feaa406c810 0x7feaa406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.706+0000 7feab6ffd640 1 --2- 172.21.15.148:0/3168826244 >> v2:172.21.15.148:6800/3205090179 conn(0x7feaa406c810 0x7feaa406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.706+0000 7feab6ffd640 10 monclient: get_auth_request con 0x7feaa406c810 auth_method 0 2023-10-30T12:39:53.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.706+0000 7feab6ffd640 1 --2- 172.21.15.148:0/3168826244 >> v2:172.21.15.148:6800/3205090179 conn(0x7feaa406c810 0x7feaa406ecd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7feab813fa80 tx=0x7feaa8007b90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.707+0000 7fea9ffff640 10 client.16314.objecter ms_handle_connect 0x7feaa406c810 2023-10-30T12:39:53.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 1 -- 172.21.15.148:0/3168826244 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(900..900 src has 739..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7feaac0010c0 con 0x7feab8065d20 2023-10-30T12:39:53.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 10 client.16314.objecter ms_dispatch 0x7feab8000b90 osd_map(900..900 src has 739..900) v4 2023-10-30T12:39:53.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 3 client.16314.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:53.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 3 client.16314.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:53.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 10 client.16314.objecter checking command 1 2023-10-30T12:39:53.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 1 --2- 172.21.15.148:0/3168826244 >> v2:172.21.15.150:6812/3463178370 conn(0x7feaa4073b30 0x7feaa4075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 20 client.16314.objecter _get_session s=0x7feaac0039e0 osd=7 3 2023-10-30T12:39:53.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 20 client.16314.objecter put_session s=0x7feaac0039e0 osd=7 3 2023-10-30T12:39:53.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 15 client.16314.objecter _session_command_op_remove -1 1 2023-10-30T12:39:53.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 20 client.16314.objecter _get_session s=0x7feaac0039e0 osd=7 3 2023-10-30T12:39:53.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 20 client.16314.objecter get_session s=0x7feaac0039e0 osd=7 3 2023-10-30T12:39:53.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 15 client.16314.objecter _session_command_op_assign 7 1 2023-10-30T12:39:53.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 20 client.16314.objecter put_session s=0x7feaac0039e0 osd=7 4 2023-10-30T12:39:53.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 10 client.16314.objecter _send_command 1 2023-10-30T12:39:53.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 1 -- 172.21.15.148:0/3168826244 --> v2:172.21.15.150:6812/3463178370 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feaac0035f0 con 0x7feaa4073b30 2023-10-30T12:39:53.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 20 client.16314.objecter dump_active .. 0 homeless 2023-10-30T12:39:53.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 1 -- 172.21.15.148:0/3168826244 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7feaac053eb0 con 0x7feab8065d20 2023-10-30T12:39:53.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.709+0000 7fea9ffff640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:39:53.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.710+0000 7feab7fff640 1 --2- 172.21.15.148:0/3168826244 >> v2:172.21.15.150:6812/3463178370 conn(0x7feaa4073b30 0x7feaa4075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.710+0000 7feab7fff640 10 monclient: get_auth_request con 0x7feaa4073b30 auth_method 0 2023-10-30T12:39:53.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.713+0000 7feab7fff640 1 --2- 172.21.15.148:0/3168826244 >> v2:172.21.15.150:6812/3463178370 conn(0x7feaa4073b30 0x7feaa4075fd0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.713+0000 7fea9ffff640 10 client.16314.objecter ms_handle_connect 0x7feaa4073b30 2023-10-30T12:39:53.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.735+0000 7fea9ffff640 1 -- 172.21.15.148:0/3168826244 <== osd.7 v2:172.21.15.150:6812/3463178370 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7feaac0035f0 con 0x7feaa4073b30 2023-10-30T12:39:53.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.735+0000 7fea9ffff640 10 client.16314.objecter ms_dispatch 0x7feab8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:39:53.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.735+0000 7fea9ffff640 10 client.16314.objecter _finish_command 1 = system:0 2023-10-30T12:39:53.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.735+0000 7fea9ffff640 20 client.16314.objecter put_session s=0x7feaac0039e0 osd=7 4 2023-10-30T12:39:53.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.735+0000 7fea9ffff640 15 client.16314.objecter _session_command_op_remove 7 1 2023-10-30T12:39:53.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 10 client.16314.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:39:53.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 15 client.16314.objecter _session_command_op_assign -1 2 2023-10-30T12:39:53.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 20 client.16314.objecter _get_session s=0x7feaac0039e0 osd=7 3 2023-10-30T12:39:53.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 20 client.16314.objecter put_session s=0x7feaac0039e0 osd=7 3 2023-10-30T12:39:53.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 20 client.16314.objecter _get_session s=0x7feaac0039e0 osd=7 3 2023-10-30T12:39:53.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 15 client.16314.objecter _session_command_op_remove -1 2 2023-10-30T12:39:53.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 20 client.16314.objecter get_session s=0x7feaac0039e0 osd=7 3 2023-10-30T12:39:53.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 15 client.16314.objecter _session_command_op_assign 7 2 2023-10-30T12:39:53.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 20 client.16314.objecter put_session s=0x7feaac0039e0 osd=7 4 2023-10-30T12:39:53.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 10 client.16314.objecter _send_command 2 2023-10-30T12:39:53.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.748+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 --> v2:172.21.15.150:6812/3463178370 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7feab8066100 con 0x7feaa4073b30 2023-10-30T12:39:53.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.761+0000 7fea9ffff640 1 -- 172.21.15.148:0/3168826244 <== osd.7 v2:172.21.15.150:6812/3463178370 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7feab0034440 con 0x7feaa4073b30 2023-10-30T12:39:53.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.761+0000 7fea9ffff640 10 client.16314.objecter ms_dispatch 0x7feab8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:39:53.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.761+0000 7fea9ffff640 10 client.16314.objecter _finish_command 2 = system:0 2023-10-30T12:39:53.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.761+0000 7fea9ffff640 20 client.16314.objecter put_session s=0x7feaac0039e0 osd=7 4 2023-10-30T12:39:53.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.761+0000 7fea9ffff640 15 client.16314.objecter _session_command_op_remove 7 2 2023-10-30T12:39:53.764 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T12:39:53.764 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:39:53.764 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T12:39:53.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 10 librados: watch_flush enter 2023-10-30T12:39:53.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 10 librados: watch_flush exit 2023-10-30T12:39:53.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 10 client.16314.objecter close_session for osd.7 2023-10-30T12:39:53.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 >> v2:172.21.15.150:6812/3463178370 conn(0x7feaa4073b30 msgr2=0x7feaa4075fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 --2- 172.21.15.148:0/3168826244 >> v2:172.21.15.150:6812/3463178370 conn(0x7feaa4073b30 0x7feaa4075fd0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 20 client.16314.objecter put_session s=0x7feaac0039e0 osd=7 1 2023-10-30T12:39:53.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 20 client.16314.objecter shutdown clearing up homeless session... 2023-10-30T12:39:53.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 10 client.16314.objecter successfully canceled tick 2023-10-30T12:39:53.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 >> v2:172.21.15.148:6800/3205090179 conn(0x7feaa406c810 msgr2=0x7feaa406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 --2- 172.21.15.148:0/3168826244 >> v2:172.21.15.148:6800/3205090179 conn(0x7feaa406c810 0x7feaa406ecd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7feab813fa80 tx=0x7feaa8007b90 comp rx=0 tx=0).stop 2023-10-30T12:39:53.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 10 monclient: shutdown 2023-10-30T12:39:53.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:53.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feab8065d20 msgr2=0x7feab81d8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 --2- 172.21.15.148:0/3168826244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feab8065d20 0x7feab81d8090 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 crypto rx=0x7feaac00c410 tx=0x7feaac00c8d0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feab7fff640 1 -- 172.21.15.148:0/3168826244 reap_dead start 2023-10-30T12:39:53.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 shutdown_connections 2023-10-30T12:39:53.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 >> 172.21.15.148:0/3168826244 conn(0x7feab806a1e0 msgr2=0x7feab80b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:53.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 shutdown_connections 2023-10-30T12:39:53.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 -- 172.21.15.148:0/3168826244 wait complete. 2023-10-30T12:39:53.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.762+0000 7feabddb0640 1 librados: shutdown 2023-10-30T12:39:53.774 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-30T12:39:53.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.881+0000 7f2503283640 10 monclient: get_monmap_and_config 2023-10-30T12:39:53.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.881+0000 7f2503283640 10 monclient: build_initial_monmap 2023-10-30T12:39:53.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.881+0000 7f2503283640 10 monclient: monmap: 2023-10-30T12:39:53.883 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:53.883 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.884 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:53.882146+0000 2023-10-30T12:39:53.884 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:53.882146+0000 2023-10-30T12:39:53.884 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:53.884 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.884 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:53.885 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:53.885 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:53.885 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:53.885 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:53.885 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.881+0000 7f2503283640 1 Processor -- start 2023-10-30T12:39:53.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.882+0000 7f2503283640 1 -- start start 2023-10-30T12:39:53.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.882+0000 7f2503283640 10 monclient: init 2023-10-30T12:39:53.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.882+0000 7f2503283640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:53.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.882+0000 7f2503283640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-10-30T12:39:53.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.882+0000 7f2503283640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24fc0b7c10 addr v1:172.21.15.148:6789/0 2023-10-30T12:39:53.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.882+0000 7f2503283640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24fc141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:39:53.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.882+0000 7f2503283640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f24fc14a460 0x7f24fc14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.883+0000 7f2503283640 10 monclient(hunting): picked mon.noname-c con 0x7f24fc14a460 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:53.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.883+0000 7f2503283640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24fc0a9350 con 0x7f24fc0b7c10 2023-10-30T12:39:53.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.883+0000 7f2503283640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24fc0aa6a0 con 0x7f24fc141940 2023-10-30T12:39:53.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.883+0000 7f2503283640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.883+0000 7f2503283640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24fc0670a0 con 0x7f24fc14a460 2023-10-30T12:39:53.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.883+0000 7f2503283640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:53.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.883+0000 7f2503283640 10 monclient(hunting): authenticate will time out at 2886.087979s 2023-10-30T12:39:53.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.900+0000 7f2501820640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f24fc14a460 0x7f24fc14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.901+0000 7f250081e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f24fc141940 0x7f24fc141d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:45368/0 (socket says 172.21.15.148:45368) 2023-10-30T12:39:53.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.901+0000 7f250081e640 1 -- 172.21.15.148:0/1312716130 learned_addr learned my addr 172.21.15.148:0/1312716130 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:39:53.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.901+0000 7f2501820640 10 monclient(hunting): get_auth_request con 0x7f24fc14a460 auth_method 0 2023-10-30T12:39:53.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.901+0000 7f2501820640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:53.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.901+0000 7f2501820640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:53.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.901+0000 7f2501820640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:53.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.902+0000 7f2501820640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:53.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.902+0000 7f2501820640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:53.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.902+0000 7f2501820640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:53.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.905+0000 7f2501820640 10 monclient(hunting): handle_auth_done global_id 22138 payload 274 2023-10-30T12:39:53.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.905+0000 7f2501820640 1 -- 172.21.15.148:0/1312716130 >> v1:172.21.15.150:6789/0 conn(0x7f24fc141940 legacy=0x7f24fc141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.905+0000 7f2501820640 1 -- 172.21.15.148:0/1312716130 >> v1:172.21.15.148:6789/0 conn(0x7f24fc0b7c10 legacy=0x7f24fc0a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:53.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.905+0000 7f2501820640 10 monclient: _finish_hunting 0 2023-10-30T12:39:53.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.905+0000 7f2501820640 1 monclient: found mon.noname-c 2023-10-30T12:39:53.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.905+0000 7f2501820640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:53.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.905+0000 7f2501820640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2023-10-30T12:39:53.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.905+0000 7f2501820640 1 -- 172.21.15.148:0/1312716130 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc0aa910 con 0x7f24fc14a460 2023-10-30T12:39:53.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.908+0000 7f2501820640 1 --2- 172.21.15.148:0/1312716130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc14a460 0x7f24fc14c850 secure :-1 s=READY pgs=5454 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009d90 tx=0x7f24f402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=131b17ca6301d549 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 1 -- 172.21.15.148:0/1312716130 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f402cbd0 con 0x7f24fc14a460 2023-10-30T12:39:53.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:53.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 10 monclient: dump: 2023-10-30T12:39:53.918 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.918 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.919 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.919 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.919 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.919 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.919 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.920 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.920 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.920 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 10 monclient: _finish_auth 0 2023-10-30T12:39:53.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:39:53.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:53.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 1 -- 172.21.15.148:0/1312716130 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f402cd40 con 0x7f24fc14a460 2023-10-30T12:39:53.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f2503283640 5 monclient: authenticate success, global_id 22138 2023-10-30T12:39:53.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f2503283640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:39:53.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:53.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f2503283640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:53.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 1 -- 172.21.15.148:0/1312716130 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f4034660 con 0x7f24fc14a460 2023-10-30T12:39:53.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f2503283640 4 set_mon_vals no callback set 2023-10-30T12:39:53.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:39:53.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.916+0000 7f24f3fff640 10 monclient: dump: 2023-10-30T12:39:53.923 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.923 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.923 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.923 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.923 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.923 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.924 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.924 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.924 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.924 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.917+0000 7f2503283640 10 monclient: shutdown 2023-10-30T12:39:53.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.917+0000 7f2503283640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:53.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.917+0000 7f2503283640 1 -- 172.21.15.148:0/1312716130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc14a460 msgr2=0x7f24fc14c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.917+0000 7f2503283640 1 --2- 172.21.15.148:0/1312716130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc14a460 0x7f24fc14c850 secure :-1 s=READY pgs=5454 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009d90 tx=0x7f24f402be20 comp rx=0 tx=0).stop 2023-10-30T12:39:53.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.917+0000 7f2503283640 1 -- 172.21.15.148:0/1312716130 shutdown_connections 2023-10-30T12:39:53.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.917+0000 7f2503283640 1 --2- 172.21.15.148:0/1312716130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc14a460 0x7f24fc14c850 unknown :-1 s=CLOSED pgs=5454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.917+0000 7f2503283640 1 -- 172.21.15.148:0/1312716130 >> 172.21.15.148:0/1312716130 conn(0x7f24fc06c660 msgr2=0x7f24fc0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:53.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.917+0000 7f2503283640 1 -- 172.21.15.148:0/1312716130 shutdown_connections 2023-10-30T12:39:53.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.917+0000 7f2503283640 1 -- 172.21.15.148:0/1312716130 wait complete. 2023-10-30T12:39:53.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.918+0000 7f2503283640 10 monclient: build_initial_monmap 2023-10-30T12:39:53.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.918+0000 7f2503283640 10 monclient: monmap: 2023-10-30T12:39:53.929 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:53.929 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:53.929 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:53.930 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:53.930 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:53.930 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:53.930 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:53.930 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:53.931 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:53.931 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.918+0000 7f2503283640 1 librados: starting msgr at 2023-10-30T12:39:53.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.918+0000 7f2503283640 1 librados: starting objecter 2023-10-30T12:39:53.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.918+0000 7f2503283640 1 Processor -- start 2023-10-30T12:39:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 1 -- start start 2023-10-30T12:39:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 1 librados: setting wanted keys 2023-10-30T12:39:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 1 librados: calling monclient init 2023-10-30T12:39:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient: init 2023-10-30T12:39:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:39:53.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc0b7c10 0x7f24fc0ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient(hunting): picked mon.noname-b con 0x7f24fc0b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:53.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f24fc141940 0x7f24fc0aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient(hunting): picked mon.noname-c con 0x7f24fc141940 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:53.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f24fc14a460 0x7f24fc0ab330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient(hunting): picked mon.noname-a con 0x7f24fc14a460 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:53.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc0b3360 con 0x7f24fc14a460 2023-10-30T12:39:53.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc0ab870 con 0x7f24fc0b7c10 2023-10-30T12:39:53.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:53.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc0afbe0 con 0x7f24fc141940 2023-10-30T12:39:53.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:53.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.919+0000 7f2503283640 10 monclient(hunting): authenticate will time out at 2886.124839s 2023-10-30T12:39:53.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.920+0000 7f250081e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f24fc141940 0x7f24fc0aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.921+0000 7f250081e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f24fc141940 0x7f24fc0aadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:51640/0 (socket says 172.21.15.148:51640) 2023-10-30T12:39:53.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.921+0000 7f250081e640 1 -- 172.21.15.148:0/2192493879 learned_addr learned my addr 172.21.15.148:0/2192493879 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:53.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.921+0000 7f250081e640 10 monclient(hunting): get_auth_request con 0x7f24fc141940 auth_method 0 2023-10-30T12:39:53.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.921+0000 7f250081e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:53.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.921+0000 7f250081e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:53.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.921+0000 7f250081e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:53.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.921+0000 7f250081e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:53.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.921+0000 7f250081e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:53.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.921+0000 7f250081e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:53.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.922+0000 7f250101f640 1 --2- 172.21.15.148:0/2192493879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc0b7c10 0x7f24fc0ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.924+0000 7f250081e640 10 monclient(hunting): handle_auth_done global_id 17765 payload 824 2023-10-30T12:39:53.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.924+0000 7f250081e640 1 -- 172.21.15.148:0/2192493879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc0b7c10 msgr2=0x7f24fc0ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.924+0000 7f250081e640 1 --2- 172.21.15.148:0/2192493879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc0b7c10 0x7f24fc0ad700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.924+0000 7f250081e640 1 -- 172.21.15.148:0/2192493879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f24fc14a460 msgr2=0x7f24fc0ab330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.924+0000 7f250081e640 1 --2- 172.21.15.148:0/2192493879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f24fc14a460 0x7f24fc0ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.924+0000 7f250081e640 10 monclient: _finish_hunting 0 2023-10-30T12:39:53.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.924+0000 7f250081e640 1 monclient: found mon.noname-c 2023-10-30T12:39:53.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.924+0000 7f250081e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:53.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.924+0000 7f250081e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:39:53.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.924+0000 7f250081e640 1 -- 172.21.15.148:0/2192493879 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc0ae380 con 0x7f24fc141940 2023-10-30T12:39:53.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.926+0000 7f250081e640 1 --2- 172.21.15.148:0/2192493879 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f24fc141940 0x7f24fc0aadf0 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00c4a0 tx=0x7f24ec00c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.926+0000 7f24f1ffb640 10 client.?.objecter ms_handle_connect 0x7f24fc141940 2023-10-30T12:39:53.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.926+0000 7f24f1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:53.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 1 -- 172.21.15.148:0/2192493879 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec005ce0 con 0x7f24fc141940 2023-10-30T12:39:53.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:53.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 10 monclient: dump: 2023-10-30T12:39:53.942 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.943 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.943 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.943 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.943 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.943 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.944 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.944 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.944 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.944 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:39:53.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:39:53.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:53.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 1 -- 172.21.15.148:0/2192493879 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ec00e040 con 0x7f24fc141940 2023-10-30T12:39:53.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f2503283640 5 monclient: authenticate success, global_id 17765 2023-10-30T12:39:53.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f2503283640 10 monclient: _renew_subs 2023-10-30T12:39:53.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f2503283640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:53.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f2503283640 1 -- 172.21.15.148:0/2192493879 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24fc0adca0 con 0x7f24fc141940 2023-10-30T12:39:53.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:53.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 1 -- 172.21.15.148:0/2192493879 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec005e90 con 0x7f24fc141940 2023-10-30T12:39:53.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:53.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:53.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f1ffb640 10 monclient: dump: 2023-10-30T12:39:53.947 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:53.947 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:53.947 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.947 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:53.948 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:53.948 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:53.948 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:53.948 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:53.948 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:53.948 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:53.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f24f37fe640 4 set_mon_vals no callback set 2023-10-30T12:39:53.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f2503283640 10 client.17765.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:53.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f2503283640 10 monclient: _renew_subs 2023-10-30T12:39:53.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f2503283640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:53.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f2503283640 1 -- 172.21.15.148:0/2192493879 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24fc0adec0 con 0x7f24fc141940 2023-10-30T12:39:53.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.930+0000 7f2503283640 1 librados: init done 2023-10-30T12:39:53.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.932+0000 7f2503283640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:53.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.932+0000 7f2503283640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:53.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.932+0000 7f2503283640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:53.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:53.932+0000 7f2503283640 1 -- 172.21.15.148:0/2192493879 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24fc075550 con 0x7f24fc141940 2023-10-30T12:39:54.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.232+0000 7f24f1ffb640 1 -- 172.21.15.148:0/2192493879 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f24ec007270 con 0x7f24fc141940 2023-10-30T12:39:54.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.233+0000 7f24f1ffb640 1 --2- 172.21.15.148:0/2192493879 >> v2:172.21.15.148:6800/3205090179 conn(0x7f24e006c7f0 0x7f24e006ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:54.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.233+0000 7f250101f640 1 --2- 172.21.15.148:0/2192493879 >> v2:172.21.15.148:6800/3205090179 conn(0x7f24e006c7f0 0x7f24e006ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:54.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.233+0000 7f250101f640 10 monclient: get_auth_request con 0x7f24e006c7f0 auth_method 0 2023-10-30T12:39:54.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.233+0000 7f250101f640 1 --2- 172.21.15.148:0/2192493879 >> v2:172.21.15.148:6800/3205090179 conn(0x7f24e006c7f0 0x7f24e006ecb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f24f8003d90 tx=0x7f24f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:54.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.233+0000 7f24f1ffb640 10 client.17765.objecter ms_handle_connect 0x7f24e006c7f0 2023-10-30T12:39:54.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.246+0000 7f24f1ffb640 1 -- 172.21.15.148:0/2192493879 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(900..900 src has 739..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f24ec057750 con 0x7f24fc141940 2023-10-30T12:39:54.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.246+0000 7f24f1ffb640 10 client.17765.objecter ms_dispatch 0x7f24fc000b90 osd_map(900..900 src has 739..900) v4 2023-10-30T12:39:54.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.246+0000 7f24f1ffb640 3 client.17765.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:54.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.246+0000 7f24f1ffb640 3 client.17765.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:54.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.246+0000 7f24f1ffb640 20 client.17765.objecter dump_active .. 0 homeless 2023-10-30T12:39:54.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.246+0000 7f24f1ffb640 1 -- 172.21.15.148:0/2192493879 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f24ec05b7f0 con 0x7f24fc141940 2023-10-30T12:39:54.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.246+0000 7f24f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:54.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.246+0000 7f24f1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:54.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.420+0000 7f2503283640 1 -- 172.21.15.148:0/2192493879 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f24fc0adca0 con 0x7f24e006c7f0 2023-10-30T12:39:54.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.421+0000 7f24f1ffb640 1 -- 172.21.15.148:0/2192493879 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f24fc0adca0 con 0x7f24e006c7f0 2023-10-30T12:39:54.423 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 7 to deep-scrub 2023-10-30T12:39:54.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.422+0000 7f2503283640 10 librados: watch_flush enter 2023-10-30T12:39:54.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.422+0000 7f2503283640 10 librados: watch_flush exit 2023-10-30T12:39:54.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.422+0000 7f2503283640 20 client.17765.objecter shutdown clearing up homeless session... 2023-10-30T12:39:54.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.422+0000 7f2503283640 10 client.17765.objecter successfully canceled tick 2023-10-30T12:39:54.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 -- 172.21.15.148:0/2192493879 >> v2:172.21.15.148:6800/3205090179 conn(0x7f24e006c7f0 msgr2=0x7f24e006ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:54.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 --2- 172.21.15.148:0/2192493879 >> v2:172.21.15.148:6800/3205090179 conn(0x7f24e006c7f0 0x7f24e006ecb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f24f8003d90 tx=0x7f24f8009040 comp rx=0 tx=0).stop 2023-10-30T12:39:54.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 10 monclient: shutdown 2023-10-30T12:39:54.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:54.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 -- 172.21.15.148:0/2192493879 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f24fc141940 msgr2=0x7f24fc0aadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:54.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 --2- 172.21.15.148:0/2192493879 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f24fc141940 0x7f24fc0aadf0 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00c4a0 tx=0x7f24ec00c960 comp rx=0 tx=0).stop 2023-10-30T12:39:54.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 -- 172.21.15.148:0/2192493879 shutdown_connections 2023-10-30T12:39:54.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 --2- 172.21.15.148:0/2192493879 >> v2:172.21.15.148:6800/3205090179 conn(0x7f24e006c7f0 0x7f24e006ecb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 --2- 172.21.15.148:0/2192493879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f24fc14a460 0x7f24fc0ab330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 --2- 172.21.15.148:0/2192493879 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f24fc141940 0x7f24fc0aadf0 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 --2- 172.21.15.148:0/2192493879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc0b7c10 0x7f24fc0ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 -- 172.21.15.148:0/2192493879 >> 172.21.15.148:0/2192493879 conn(0x7f24fc06c660 msgr2=0x7f24fc13b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:54.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 -- 172.21.15.148:0/2192493879 shutdown_connections 2023-10-30T12:39:54.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 -- 172.21.15.148:0/2192493879 wait complete. 2023-10-30T12:39:54.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.423+0000 7f2503283640 1 librados: shutdown 2023-10-30T12:39:54.436 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:39:54.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.536+0000 7f0bfa20e640 10 monclient: get_monmap_and_config 2023-10-30T12:39:54.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.536+0000 7f0bfa20e640 10 monclient: build_initial_monmap 2023-10-30T12:39:54.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.536+0000 7f0bfa20e640 10 monclient: monmap: 2023-10-30T12:39:54.539 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:54.539 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:54.539 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:39:54.538014+0000 2023-10-30T12:39:54.540 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:39:54.538014+0000 2023-10-30T12:39:54.540 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:54.540 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:54.540 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:39:54.540 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:39:54.541 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:39:54.541 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:39:54.541 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:39:54.541 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:54.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.537+0000 7f0bfa20e640 1 Processor -- start 2023-10-30T12:39:54.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.537+0000 7f0bfa20e640 1 -- start start 2023-10-30T12:39:54.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.537+0000 7f0bfa20e640 10 monclient: init 2023-10-30T12:39:54.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:54.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-10-30T12:39:54.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf406a0e0 0x7f0bf40aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:54.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf406a0e0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:54.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0bf413ed20 0x7f0bf413f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:54.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf413ed20 addr v2:172.21.15.148:3300/0 2023-10-30T12:39:54.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0bf4141ba0 0x7f0bf4141fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:54.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 10 monclient(hunting): picked mon.noname-c con 0x7f0bf4141ba0 addr v2:172.21.15.150:3300/0 2023-10-30T12:39:54.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:54.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf4067340 con 0x7f0bf413ed20 2023-10-30T12:39:54.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:54.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf406a4c0 con 0x7f0bf4141ba0 2023-10-30T12:39:54.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:54.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40aeb80 con 0x7f0bf406a0e0 2023-10-30T12:39:54.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:54.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bfa20e640 10 monclient(hunting): authenticate will time out at 2886.743246s 2023-10-30T12:39:54.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf406a0e0 0x7f0bf40aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:54.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.538+0000 7f0bf2ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0bf413ed20 0x7f0bf413f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:54.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf2ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0bf413ed20 0x7f0bf413f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52398/0 (socket says 172.21.15.148:52398) 2023-10-30T12:39:54.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf2ffd640 1 -- 172.21.15.148:0/267726692 learned_addr learned my addr 172.21.15.148:0/267726692 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:54.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf2ffd640 10 monclient(hunting): get_auth_request con 0x7f0bf413ed20 auth_method 0 2023-10-30T12:39:54.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:54.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:54.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:54.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf37fe640 10 monclient(hunting): get_auth_request con 0x7f0bf406a0e0 auth_method 0 2023-10-30T12:39:54.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:54.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:54.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:54.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf3fff640 1 --2- 172.21.15.148:0/267726692 >> v2:172.21.15.150:3300/0 conn(0x7f0bf4141ba0 0x7f0bf4141fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:54.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf3fff640 10 monclient(hunting): get_auth_request con 0x7f0bf4141ba0 auth_method 0 2023-10-30T12:39:54.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:54.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:54.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:54.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:54.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:54.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:54.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:54.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:54.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.539+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:54.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.540+0000 7f0bf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:54.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.540+0000 7f0bf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:54.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.540+0000 7f0bf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:54.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_done global_id 17768 payload 274 2023-10-30T12:39:54.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf37fe640 1 -- 172.21.15.148:0/267726692 >> v2:172.21.15.150:3300/0 conn(0x7f0bf4141ba0 msgr2=0x7f0bf4141fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:54.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf37fe640 1 --2- 172.21.15.148:0/267726692 >> v2:172.21.15.150:3300/0 conn(0x7f0bf4141ba0 0x7f0bf4141fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf37fe640 1 -- 172.21.15.148:0/267726692 >> v2:172.21.15.148:3300/0 conn(0x7f0bf413ed20 msgr2=0x7f0bf413f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:54.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf37fe640 1 --2- 172.21.15.148:0/267726692 >> v2:172.21.15.148:3300/0 conn(0x7f0bf413ed20 0x7f0bf413f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf37fe640 10 monclient: _finish_hunting 0 2023-10-30T12:39:54.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf37fe640 1 monclient: found mon.noname-b 2023-10-30T12:39:54.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:54.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.148:3301/0 2023-10-30T12:39:54.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf37fe640 1 -- 172.21.15.148:0/267726692 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf413f6a0 con 0x7f0bf406a0e0 2023-10-30T12:39:54.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.541+0000 7f0bf2ffd640 1 --2- 172.21.15.148:0/267726692 >> v2:172.21.15.148:3300/0 conn(0x7f0bf413ed20 0x7f0bf413f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:39:54.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.543+0000 7f0bf37fe640 1 --2- 172.21.15.148:0/267726692 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf406a0e0 0x7f0bf40aef40 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40aed00 tx=0x7f0be402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3293ca2c3f56306 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:54.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 1 -- 172.21.15.148:0/267726692 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4031830 con 0x7f0bf406a0e0 2023-10-30T12:39:54.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:54.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:39:54.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 10 monclient: dump: 2023-10-30T12:39:54.560 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:54.560 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:54.560 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:54.560 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:54.561 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:54.561 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:54.561 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:54.561 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:54.561 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:54.561 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:54.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 10 monclient: _finish_auth 0 2023-10-30T12:39:54.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:39:54.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:54.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 1 -- 172.21.15.148:0/267726692 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be402d040 con 0x7f0bf406a0e0 2023-10-30T12:39:54.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:54.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 1 -- 172.21.15.148:0/267726692 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4031bc0 con 0x7f0bf406a0e0 2023-10-30T12:39:54.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bfa20e640 5 monclient: authenticate success, global_id 17768 2023-10-30T12:39:54.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bfa20e640 10 monclient: get_monmap_and_config success 2023-10-30T12:39:54.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bfa20e640 4 set_mon_vals no callback set 2023-10-30T12:39:54.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:54.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:54.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.546+0000 7f0bf27fc640 10 monclient: dump: 2023-10-30T12:39:54.564 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:54.564 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:54.564 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:54.564 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:54.565 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:54.565 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:54.565 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:54.565 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:54.565 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:54.565 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:54.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 10 monclient: shutdown 2023-10-30T12:39:54.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:54.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 1 -- 172.21.15.148:0/267726692 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf406a0e0 msgr2=0x7f0bf40aef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:54.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 1 --2- 172.21.15.148:0/267726692 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf406a0e0 0x7f0bf40aef40 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40aed00 tx=0x7f0be402b8d0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 1 -- 172.21.15.148:0/267726692 shutdown_connections 2023-10-30T12:39:54.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 1 --2- 172.21.15.148:0/267726692 >> v2:172.21.15.148:3300/0 conn(0x7f0bf413ed20 0x7f0bf413f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 1 --2- 172.21.15.148:0/267726692 >> v2:172.21.15.150:3300/0 conn(0x7f0bf4141ba0 0x7f0bf4141fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 1 --2- 172.21.15.148:0/267726692 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf406a0e0 0x7f0bf40aef40 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 1 -- 172.21.15.148:0/267726692 >> 172.21.15.148:0/267726692 conn(0x7f0bf406b0b0 msgr2=0x7f0bf406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:54.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 1 -- 172.21.15.148:0/267726692 shutdown_connections 2023-10-30T12:39:54.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.547+0000 7f0bfa20e640 1 -- 172.21.15.148:0/267726692 wait complete. 2023-10-30T12:39:54.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.548+0000 7f0bfa20e640 10 monclient: build_initial_monmap 2023-10-30T12:39:54.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.548+0000 7f0bfa20e640 10 monclient: monmap: 2023-10-30T12:39:54.568 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:39:54.568 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:39:54.569 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:39:54.569 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:39:54.569 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:39:54.569 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:39:54.569 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:39:54.569 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:39:54.570 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:39:54.570 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:54.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.548+0000 7f0bfa20e640 1 librados: starting msgr at 2023-10-30T12:39:54.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.548+0000 7f0bfa20e640 1 librados: starting objecter 2023-10-30T12:39:54.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.548+0000 7f0bfa20e640 1 Processor -- start 2023-10-30T12:39:54.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.548+0000 7f0bfa20e640 1 -- start start 2023-10-30T12:39:54.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.548+0000 7f0bfa20e640 1 librados: setting wanted keys 2023-10-30T12:39:54.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.548+0000 7f0bfa20e640 1 librados: calling monclient init 2023-10-30T12:39:54.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.548+0000 7f0bfa20e640 10 monclient: init 2023-10-30T12:39:54.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 10 monclient: _reopen_session rank -1 2023-10-30T12:39:54.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:39:54.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bf406a0e0 0x7f0bf40713e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:54.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf406a0e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:39:54.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf413ed20 0x7f0bf41370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:54.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 10 monclient(hunting): picked mon.noname-c con 0x7f0bf413ed20 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:39:54.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf4141ba0 0x7f0bf41375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:54.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf4141ba0 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:39:54.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:54.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40ad960 con 0x7f0bf4141ba0 2023-10-30T12:39:54.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:54.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf41431b0 con 0x7f0bf406a0e0 2023-10-30T12:39:54.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:39:54.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf4142d20 con 0x7f0bf413ed20 2023-10-30T12:39:54.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 10 monclient(hunting): _renew_subs 2023-10-30T12:39:54.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bfa20e640 10 monclient(hunting): authenticate will time out at 2886.754355s 2023-10-30T12:39:54.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bf2ffd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf413ed20 0x7f0bf41370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:54.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.549+0000 7f0bf3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf4141ba0 0x7f0bf41375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:54.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bf406a0e0 0x7f0bf40713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:54.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf2ffd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf413ed20 0x7f0bf41370a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:51670/0 (socket says 172.21.15.148:51670) 2023-10-30T12:39:54.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf2ffd640 1 -- 172.21.15.148:0/3989294004 learned_addr learned my addr 172.21.15.148:0/3989294004 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:39:54.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf2ffd640 10 monclient(hunting): get_auth_request con 0x7f0bf413ed20 auth_method 0 2023-10-30T12:39:54.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:54.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:54.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:54.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf3fff640 10 monclient(hunting): get_auth_request con 0x7f0bf4141ba0 auth_method 0 2023-10-30T12:39:54.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:54.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:54.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:54.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf37fe640 10 monclient(hunting): get_auth_request con 0x7f0bf406a0e0 auth_method 0 2023-10-30T12:39:54.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:39:54.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:39:54.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:39:54.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:54.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:54.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:54.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:54.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:54.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.550+0000 7f0bf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:54.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.551+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:39:54.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.551+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:39:54.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.551+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:39:54.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.553+0000 7f0bf2ffd640 10 monclient(hunting): handle_auth_done global_id 17771 payload 824 2023-10-30T12:39:54.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.553+0000 7f0bf2ffd640 1 -- 172.21.15.148:0/3989294004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bf406a0e0 msgr2=0x7f0bf40713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:54.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.553+0000 7f0bf2ffd640 1 --2- 172.21.15.148:0/3989294004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bf406a0e0 0x7f0bf40713e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.553+0000 7f0bf2ffd640 1 -- 172.21.15.148:0/3989294004 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf4141ba0 msgr2=0x7f0bf41375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:54.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.553+0000 7f0bf2ffd640 1 --2- 172.21.15.148:0/3989294004 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf4141ba0 0x7f0bf41375e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:54.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.553+0000 7f0bf2ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:39:54.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.553+0000 7f0bf2ffd640 1 monclient: found mon.noname-c 2023-10-30T12:39:54.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.553+0000 7f0bf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:39:54.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.553+0000 7f0bf2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.148:3301/0 2023-10-30T12:39:54.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.553+0000 7f0bf2ffd640 1 -- 172.21.15.148:0/3989294004 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf4070210 con 0x7f0bf413ed20 2023-10-30T12:39:54.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.556+0000 7f0bf2ffd640 1 --2- 172.21.15.148:0/3989294004 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf413ed20 0x7f0bf41370a0 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7f0be800e9b0 tx=0x7f0be800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:54.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.556+0000 7f0bdbfff640 10 client.?.objecter ms_handle_connect 0x7f0bf413ed20 2023-10-30T12:39:54.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.556+0000 7f0bdbfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:39:54.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 1 -- 172.21.15.148:0/3989294004 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8004030 con 0x7f0bf413ed20 2023-10-30T12:39:54.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:54.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:39:54.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 10 monclient: dump: 2023-10-30T12:39:54.584 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:54.584 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:54.584 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:54.585 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:54.585 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:54.585 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:54.585 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:54.585 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:54.585 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:54.586 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:54.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 10 monclient: _finish_auth 0 2023-10-30T12:39:54.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:39:54.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:39:54.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 1 -- 172.21.15.148:0/3989294004 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be800d040 con 0x7f0bf413ed20 2023-10-30T12:39:54.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:39:54.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 1 -- 172.21.15.148:0/3989294004 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be80043a0 con 0x7f0bf413ed20 2023-10-30T12:39:54.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bfa20e640 5 monclient: authenticate success, global_id 17771 2023-10-30T12:39:54.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bfa20e640 10 monclient: _renew_subs 2023-10-30T12:39:54.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bfa20e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:54.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bfa20e640 1 -- 172.21.15.148:0/3989294004 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf4137b20 con 0x7f0bf413ed20 2023-10-30T12:39:54.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:39:54.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:39:54.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bdbfff640 10 monclient: dump: 2023-10-30T12:39:54.588 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:39:54.589 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:39:54.589 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:54.589 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:39:54.589 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:39:54.589 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:39:54.589 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:39:54.590 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:39:54.590 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:39:54.590 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:39:54.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bf1ffb640 4 set_mon_vals no callback set 2023-10-30T12:39:54.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bfa20e640 10 client.17771.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:39:54.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bfa20e640 10 monclient: _renew_subs 2023-10-30T12:39:54.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bfa20e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:54.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bfa20e640 1 -- 172.21.15.148:0/3989294004 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf4144a30 con 0x7f0bf413ed20 2023-10-30T12:39:54.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.559+0000 7f0bfa20e640 1 librados: init done 2023-10-30T12:39:54.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.560+0000 7f0bfa20e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:54.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.560+0000 7f0bfa20e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:54.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.560+0000 7f0bfa20e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.148:3301/0 2023-10-30T12:39:54.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.560+0000 7f0bfa20e640 1 -- 172.21.15.148:0/3989294004 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf4075550 con 0x7f0bf413ed20 2023-10-30T12:39:54.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.579+0000 7f0bdbfff640 1 -- 172.21.15.148:0/3989294004 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f0be800c9a0 con 0x7f0bf413ed20 2023-10-30T12:39:54.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.580+0000 7f0bdbfff640 1 --2- 172.21.15.148:0/3989294004 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0be006c810 0x7f0be006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:54.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.580+0000 7f0bf37fe640 1 --2- 172.21.15.148:0/3989294004 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0be006c810 0x7f0be006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:54.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.580+0000 7f0bf37fe640 10 monclient: get_auth_request con 0x7f0be006c810 auth_method 0 2023-10-30T12:39:54.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.580+0000 7f0bf37fe640 1 --2- 172.21.15.148:0/3989294004 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0be006c810 0x7f0be006ecd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0be4036000 tx=0x7f0be403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:54.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.580+0000 7f0bdbfff640 10 client.17771.objecter ms_handle_connect 0x7f0be006c810 2023-10-30T12:39:54.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.582+0000 7f0bdbfff640 1 -- 172.21.15.148:0/3989294004 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(900..900 src has 739..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f0be8058380 con 0x7f0bf413ed20 2023-10-30T12:39:54.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.582+0000 7f0bdbfff640 10 client.17771.objecter ms_dispatch 0x7f0bf4000b90 osd_map(900..900 src has 739..900) v4 2023-10-30T12:39:54.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.582+0000 7f0bdbfff640 3 client.17771.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:39:54.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.582+0000 7f0bdbfff640 3 client.17771.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:39:54.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.582+0000 7f0bdbfff640 20 client.17771.objecter dump_active .. 0 homeless 2023-10-30T12:39:54.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.816+0000 7f0bdbfff640 1 -- 172.21.15.148:0/3989294004 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0be805c420 con 0x7f0bf413ed20 2023-10-30T12:39:54.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.816+0000 7f0bdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:39:54.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.816+0000 7f0bdbfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:39:54.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.986+0000 7f0bfa20e640 1 -- 172.21.15.148:0/3989294004 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0bf406e080 con 0x7f0be006c810 2023-10-30T12:39:54.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.990+0000 7f0bdbfff640 1 -- 172.21.15.148:0/3989294004 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151345 (secure 0 0 0) 0x7f0be403a420 con 0x7f0be006c810 2023-10-30T12:39:54.994 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:39:54.994 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T12:39:54.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 10 librados: watch_flush enter 2023-10-30T12:39:54.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 10 librados: watch_flush exit 2023-10-30T12:39:54.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 20 client.17771.objecter shutdown clearing up homeless session... 2023-10-30T12:39:54.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 10 client.17771.objecter successfully canceled tick 2023-10-30T12:39:54.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 -- 172.21.15.148:0/3989294004 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0be006c810 msgr2=0x7f0be006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:54.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 --2- 172.21.15.148:0/3989294004 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0be006c810 0x7f0be006ecd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0be4036000 tx=0x7f0be403c040 comp rx=0 tx=0).stop 2023-10-30T12:39:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 10 monclient: shutdown 2023-10-30T12:39:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:39:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 -- 172.21.15.148:0/3989294004 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf413ed20 msgr2=0x7f0bf41370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 --2- 172.21.15.148:0/3989294004 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf413ed20 0x7f0bf41370a0 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7f0be800e9b0 tx=0x7f0be800ee70 comp rx=0 tx=0).stop 2023-10-30T12:39:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 -- 172.21.15.148:0/3989294004 shutdown_connections 2023-10-30T12:39:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 --2- 172.21.15.148:0/3989294004 >> v2:172.21.15.148:6800/3205090179 conn(0x7f0be006c810 0x7f0be006ecd0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 --2- 172.21.15.148:0/3989294004 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf4141ba0 0x7f0bf41375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 --2- 172.21.15.148:0/3989294004 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0bf413ed20 0x7f0bf41370a0 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 --2- 172.21.15.148:0/3989294004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bf406a0e0 0x7f0bf40713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 -- 172.21.15.148:0/3989294004 >> 172.21.15.148:0/3989294004 conn(0x7f0bf406b0b0 msgr2=0x7f0bf406b490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 -- 172.21.15.148:0/3989294004 shutdown_connections 2023-10-30T12:39:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 -- 172.21.15.148:0/3989294004 wait complete. 2023-10-30T12:39:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:39:54.993+0000 7f0bfa20e640 1 librados: shutdown 2023-10-30T12:39:55.007 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1566,"stamp":"2023-10-30T12:39:53.311601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":543544,"kb_used_data":70488,"kb_used_omap":12,"kb_used_meta":471411,"kb_avail":754431176,"statfs":{"total":773094113280,"available":772537524224,"internally_reserved":0,"allocated":72179712,"data_stored":36200504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":482725433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":84,"apply_latency_ms":84,"commit_latency_ns":84000000,"apply_latency_ns":84000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.921197"},"pg_stats":[{"pgid":"20.8","version":"0'0","reported_seq":840,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.414734+0000","last_change":"2023-10-30T12:19:35.856437+0000","last_active":"2023-10-30T12:39:43.414734+0000","last_peered":"2023-10-30T12:39:43.414734+0000","last_clean":"2023-10-30T12:39:43.414734+0000","last_became_active":"2023-10-30T12:19:35.850666+0000","last_became_peered":"2023-10-30T12:19:35.850666+0000","last_unstale":"2023-10-30T12:39:43.414734+0000","last_undegraded":"2023-10-30T12:39:43.414734+0000","last_fullsized":"2023-10-30T12:39:43.414734+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:24.452037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":840,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.417989+0000","last_change":"2023-10-30T12:19:38.560827+0000","last_active":"2023-10-30T12:39:43.417989+0000","last_peered":"2023-10-30T12:39:43.417989+0000","last_clean":"2023-10-30T12:39:43.417989+0000","last_became_active":"2023-10-30T12:19:38.546595+0000","last_became_peered":"2023-10-30T12:19:38.546595+0000","last_unstale":"2023-10-30T12:39:43.417989+0000","last_undegraded":"2023-10-30T12:39:43.417989+0000","last_fullsized":"2023-10-30T12:39:43.417989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:22.315389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":840,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.419846+0000","last_change":"2023-10-30T12:19:35.881177+0000","last_active":"2023-10-30T12:39:43.419846+0000","last_peered":"2023-10-30T12:39:43.419846+0000","last_clean":"2023-10-30T12:39:43.419846+0000","last_became_active":"2023-10-30T12:19:35.871000+0000","last_became_peered":"2023-10-30T12:19:35.871000+0000","last_unstale":"2023-10-30T12:39:43.419846+0000","last_undegraded":"2023-10-30T12:39:43.419846+0000","last_fullsized":"2023-10-30T12:39:43.419846+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:16.108884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":845,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.461453+0000","last_change":"2023-10-30T12:19:29.602055+0000","last_active":"2023-10-30T12:39:43.461453+0000","last_peered":"2023-10-30T12:39:43.461453+0000","last_clean":"2023-10-30T12:39:43.461453+0000","last_became_active":"2023-10-30T12:19:29.594505+0000","last_became_peered":"2023-10-30T12:19:29.594505+0000","last_unstale":"2023-10-30T12:39:43.461453+0000","last_undegraded":"2023-10-30T12:39:43.461453+0000","last_fullsized":"2023-10-30T12:39:43.461453+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:15:25.873922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":849,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:44.657936+0000","last_change":"2023-10-30T12:19:27.523347+0000","last_active":"2023-10-30T12:39:44.657936+0000","last_peered":"2023-10-30T12:39:44.657936+0000","last_clean":"2023-10-30T12:39:44.657936+0000","last_became_active":"2023-10-30T12:19:27.515739+0000","last_became_peered":"2023-10-30T12:19:27.515739+0000","last_unstale":"2023-10-30T12:39:44.657936+0000","last_undegraded":"2023-10-30T12:39:44.657936+0000","last_fullsized":"2023-10-30T12:39:44.657936+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:36.852443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":838,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.854313+0000","last_change":"2023-10-30T12:19:31.914796+0000","last_active":"2023-10-30T12:39:43.854313+0000","last_peered":"2023-10-30T12:39:43.854313+0000","last_clean":"2023-10-30T12:39:43.854313+0000","last_became_active":"2023-10-30T12:19:31.894071+0000","last_became_peered":"2023-10-30T12:19:31.894071+0000","last_unstale":"2023-10-30T12:39:43.854313+0000","last_undegraded":"2023-10-30T12:39:43.854313+0000","last_fullsized":"2023-10-30T12:39:43.854313+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:22.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":837,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:44.361946+0000","last_change":"2023-10-30T12:19:28.154596+0000","last_active":"2023-10-30T12:39:44.361946+0000","last_peered":"2023-10-30T12:39:44.361946+0000","last_clean":"2023-10-30T12:39:44.361946+0000","last_became_active":"2023-10-30T12:19:28.142591+0000","last_became_peered":"2023-10-30T12:19:28.142591+0000","last_unstale":"2023-10-30T12:39:44.361946+0000","last_undegraded":"2023-10-30T12:39:44.361946+0000","last_fullsized":"2023-10-30T12:39:44.361946+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:46:05.070599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":837,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.630198+0000","last_change":"2023-10-30T12:19:38.653024+0000","last_active":"2023-10-30T12:39:43.630198+0000","last_peered":"2023-10-30T12:39:43.630198+0000","last_clean":"2023-10-30T12:39:43.630198+0000","last_became_active":"2023-10-30T12:19:38.643834+0000","last_became_peered":"2023-10-30T12:19:38.643834+0000","last_unstale":"2023-10-30T12:39:43.630198+0000","last_undegraded":"2023-10-30T12:39:43.630198+0000","last_fullsized":"2023-10-30T12:39:43.630198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:54.052947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":846,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.453262+0000","last_change":"2023-10-30T12:19:29.499316+0000","last_active":"2023-10-30T12:39:43.453262+0000","last_peered":"2023-10-30T12:39:43.453262+0000","last_clean":"2023-10-30T12:39:43.453262+0000","last_became_active":"2023-10-30T12:19:29.492043+0000","last_became_peered":"2023-10-30T12:19:29.492043+0000","last_unstale":"2023-10-30T12:39:43.453262+0000","last_undegraded":"2023-10-30T12:39:43.453262+0000","last_fullsized":"2023-10-30T12:39:43.453262+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:49.202008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1","version":"38'7","reported_seq":844,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:47.745579+0000","last_change":"2023-10-30T12:19:25.218622+0000","last_active":"2023-10-30T12:39:47.745579+0000","last_peered":"2023-10-30T12:39:47.745579+0000","last_clean":"2023-10-30T12:39:47.745579+0000","last_became_active":"2023-10-30T12:19:25.212397+0000","last_became_peered":"2023-10-30T12:19:25.212397+0000","last_unstale":"2023-10-30T12:39:47.745579+0000","last_undegraded":"2023-10-30T12:39:47.745579+0000","last_fullsized":"2023-10-30T12:39:47.745579+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":849,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.497008+0000","last_change":"2023-10-30T12:19:29.500965+0000","last_active":"2023-10-30T12:39:43.497008+0000","last_peered":"2023-10-30T12:39:43.497008+0000","last_clean":"2023-10-30T12:39:43.497008+0000","last_became_active":"2023-10-30T12:19:29.491706+0000","last_became_peered":"2023-10-30T12:19:29.491706+0000","last_unstale":"2023-10-30T12:39:43.497008+0000","last_undegraded":"2023-10-30T12:39:43.497008+0000","last_fullsized":"2023-10-30T12:39:43.497008+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:49.491590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":850,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:44.672401+0000","last_change":"2023-10-30T12:19:32.994118+0000","last_active":"2023-10-30T12:39:44.672401+0000","last_peered":"2023-10-30T12:39:44.672401+0000","last_clean":"2023-10-30T12:39:44.672401+0000","last_became_active":"2023-10-30T12:19:32.988808+0000","last_became_peered":"2023-10-30T12:19:32.988808+0000","last_unstale":"2023-10-30T12:39:44.672401+0000","last_undegraded":"2023-10-30T12:39:44.672401+0000","last_fullsized":"2023-10-30T12:39:44.672401+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:25.605041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":849,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:44.668668+0000","last_change":"2023-10-30T12:19:28.922326+0000","last_active":"2023-10-30T12:39:44.668668+0000","last_peered":"2023-10-30T12:39:44.668668+0000","last_clean":"2023-10-30T12:39:44.668668+0000","last_became_active":"2023-10-30T12:19:28.916380+0000","last_became_peered":"2023-10-30T12:19:28.916380+0000","last_unstale":"2023-10-30T12:39:44.668668+0000","last_undegraded":"2023-10-30T12:39:44.668668+0000","last_fullsized":"2023-10-30T12:39:44.668668+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:38:19.847301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":855,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.514273+0000","last_change":"2023-10-30T12:18:01.574170+0000","last_active":"2023-10-30T12:39:43.514273+0000","last_peered":"2023-10-30T12:39:43.514273+0000","last_clean":"2023-10-30T12:39:43.514273+0000","last_became_active":"2023-10-30T12:18:01.568787+0000","last_became_peered":"2023-10-30T12:18:01.568787+0000","last_unstale":"2023-10-30T12:39:43.514273+0000","last_undegraded":"2023-10-30T12:39:43.514273+0000","last_fullsized":"2023-10-30T12:39:43.514273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:12.243846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":837,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:44.345069+0000","last_change":"2023-10-30T12:19:40.949818+0000","last_active":"2023-10-30T12:39:44.345069+0000","last_peered":"2023-10-30T12:39:44.345069+0000","last_clean":"2023-10-30T12:39:44.345069+0000","last_became_active":"2023-10-30T12:19:40.940463+0000","last_became_peered":"2023-10-30T12:19:40.940463+0000","last_unstale":"2023-10-30T12:39:44.345069+0000","last_undegraded":"2023-10-30T12:39:44.345069+0000","last_fullsized":"2023-10-30T12:39:44.345069+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:43:05.246911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":843,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:44.359888+0000","last_change":"2023-10-30T12:18:01.632325+0000","last_active":"2023-10-30T12:39:44.359888+0000","last_peered":"2023-10-30T12:39:44.359888+0000","last_clean":"2023-10-30T12:39:44.359888+0000","last_became_active":"2023-10-30T12:18:01.602190+0000","last_became_peered":"2023-10-30T12:18:01.602190+0000","last_unstale":"2023-10-30T12:39:44.359888+0000","last_undegraded":"2023-10-30T12:39:44.359888+0000","last_fullsized":"2023-10-30T12:39:44.359888+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:29:50.132707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":848,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.479839+0000","last_change":"2023-10-30T12:19:40.722588+0000","last_active":"2023-10-30T12:39:43.479839+0000","last_peered":"2023-10-30T12:39:43.479839+0000","last_clean":"2023-10-30T12:39:43.479839+0000","last_became_active":"2023-10-30T12:19:40.713186+0000","last_became_peered":"2023-10-30T12:19:40.713186+0000","last_unstale":"2023-10-30T12:39:43.479839+0000","last_undegraded":"2023-10-30T12:39:43.479839+0000","last_fullsized":"2023-10-30T12:39:43.479839+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:38:31.344804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":854,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.491674+0000","last_change":"2023-10-30T12:18:01.060414+0000","last_active":"2023-10-30T12:39:43.491674+0000","last_peered":"2023-10-30T12:39:43.491674+0000","last_clean":"2023-10-30T12:39:43.491674+0000","last_became_active":"2023-10-30T12:18:01.051697+0000","last_became_peered":"2023-10-30T12:18:01.051697+0000","last_unstale":"2023-10-30T12:39:43.491674+0000","last_undegraded":"2023-10-30T12:39:43.491674+0000","last_fullsized":"2023-10-30T12:39:43.491674+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:10.773828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":830,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:47.754819+0000","last_change":"2023-10-30T12:19:40.870027+0000","last_active":"2023-10-30T12:39:47.754819+0000","last_peered":"2023-10-30T12:39:47.754819+0000","last_clean":"2023-10-30T12:39:47.754819+0000","last_became_active":"2023-10-30T12:19:40.864175+0000","last_became_peered":"2023-10-30T12:19:40.864175+0000","last_unstale":"2023-10-30T12:39:47.754819+0000","last_undegraded":"2023-10-30T12:39:47.754819+0000","last_fullsized":"2023-10-30T12:39:47.754819+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":844,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.840183+0000","last_change":"2023-10-30T12:18:01.550154+0000","last_active":"2023-10-30T12:39:43.840183+0000","last_peered":"2023-10-30T12:39:43.840183+0000","last_clean":"2023-10-30T12:39:43.840183+0000","last_became_active":"2023-10-30T12:18:01.523269+0000","last_became_peered":"2023-10-30T12:18:01.523269+0000","last_unstale":"2023-10-30T12:39:43.840183+0000","last_undegraded":"2023-10-30T12:39:43.840183+0000","last_fullsized":"2023-10-30T12:39:43.840183+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:08.405234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":837,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.838109+0000","last_change":"2023-10-30T12:19:40.449659+0000","last_active":"2023-10-30T12:39:43.838109+0000","last_peered":"2023-10-30T12:39:43.838109+0000","last_clean":"2023-10-30T12:39:43.838109+0000","last_became_active":"2023-10-30T12:19:40.440094+0000","last_became_peered":"2023-10-30T12:19:40.440094+0000","last_unstale":"2023-10-30T12:39:43.838109+0000","last_undegraded":"2023-10-30T12:39:43.838109+0000","last_fullsized":"2023-10-30T12:39:43.838109+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:12.412748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":855,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.499149+0000","last_change":"2023-10-30T12:18:01.509923+0000","last_active":"2023-10-30T12:39:43.499149+0000","last_peered":"2023-10-30T12:39:43.499149+0000","last_clean":"2023-10-30T12:39:43.499149+0000","last_became_active":"2023-10-30T12:18:01.504543+0000","last_became_peered":"2023-10-30T12:18:01.504543+0000","last_unstale":"2023-10-30T12:39:43.499149+0000","last_undegraded":"2023-10-30T12:39:43.499149+0000","last_fullsized":"2023-10-30T12:39:43.499149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:39.886008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":840,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.467788+0000","last_change":"2023-10-30T12:19:29.552208+0000","last_active":"2023-10-30T12:39:43.467788+0000","last_peered":"2023-10-30T12:39:43.467788+0000","last_clean":"2023-10-30T12:39:43.467788+0000","last_became_active":"2023-10-30T12:19:29.543275+0000","last_became_peered":"2023-10-30T12:19:29.543275+0000","last_unstale":"2023-10-30T12:39:43.467788+0000","last_undegraded":"2023-10-30T12:39:43.467788+0000","last_fullsized":"2023-10-30T12:39:43.467788+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:51.785658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":856,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:44.663833+0000","last_change":"2023-10-30T12:18:00.932461+0000","last_active":"2023-10-30T12:39:44.663833+0000","last_peered":"2023-10-30T12:39:44.663833+0000","last_clean":"2023-10-30T12:39:44.663833+0000","last_became_active":"2023-10-30T12:18:00.904525+0000","last_became_peered":"2023-10-30T12:18:00.904525+0000","last_unstale":"2023-10-30T12:39:44.663833+0000","last_undegraded":"2023-10-30T12:39:44.663833+0000","last_fullsized":"2023-10-30T12:39:44.663833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:42.734208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":830,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:47.762096+0000","last_change":"2023-10-30T12:19:26.068263+0000","last_active":"2023-10-30T12:39:47.762096+0000","last_peered":"2023-10-30T12:39:47.762096+0000","last_clean":"2023-10-30T12:39:47.762096+0000","last_became_active":"2023-10-30T12:19:26.061689+0000","last_became_peered":"2023-10-30T12:19:26.061689+0000","last_unstale":"2023-10-30T12:39:47.762096+0000","last_undegraded":"2023-10-30T12:39:47.762096+0000","last_fullsized":"2023-10-30T12:39:47.762096+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":855,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.473601+0000","last_change":"2023-10-30T12:18:01.030843+0000","last_active":"2023-10-30T12:39:43.473601+0000","last_peered":"2023-10-30T12:39:43.473601+0000","last_clean":"2023-10-30T12:39:43.473601+0000","last_became_active":"2023-10-30T12:18:01.003382+0000","last_became_peered":"2023-10-30T12:18:01.003382+0000","last_unstale":"2023-10-30T12:39:43.473601+0000","last_undegraded":"2023-10-30T12:39:43.473601+0000","last_fullsized":"2023-10-30T12:39:43.473601+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:20.711334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":830,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:47.763115+0000","last_change":"2023-10-30T12:19:25.299105+0000","last_active":"2023-10-30T12:39:47.763115+0000","last_peered":"2023-10-30T12:39:47.763115+0000","last_clean":"2023-10-30T12:39:47.763115+0000","last_became_active":"2023-10-30T12:19:25.272220+0000","last_became_peered":"2023-10-30T12:19:25.272220+0000","last_unstale":"2023-10-30T12:39:47.763115+0000","last_undegraded":"2023-10-30T12:39:47.763115+0000","last_fullsized":"2023-10-30T12:39:47.763115+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":855,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.483611+0000","last_change":"2023-10-30T12:18:01.406267+0000","last_active":"2023-10-30T12:39:43.483611+0000","last_peered":"2023-10-30T12:39:43.483611+0000","last_clean":"2023-10-30T12:39:43.483611+0000","last_became_active":"2023-10-30T12:18:01.393697+0000","last_became_peered":"2023-10-30T12:18:01.393697+0000","last_unstale":"2023-10-30T12:39:43.483611+0000","last_undegraded":"2023-10-30T12:39:43.483611+0000","last_fullsized":"2023-10-30T12:39:43.483611+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:59.043368+0000","last_clean_scrub_stamp":"2023-10-30T12:17:59.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:15:29.710122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":944,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:47.739317+0000","last_change":"2023-10-30T12:17:56.604947+0000","last_active":"2023-10-30T12:39:47.739317+0000","last_peered":"2023-10-30T12:39:47.739317+0000","last_clean":"2023-10-30T12:39:47.739317+0000","last_became_active":"2023-10-30T12:17:56.575838+0000","last_became_peered":"2023-10-30T12:17:56.575838+0000","last_unstale":"2023-10-30T12:39:47.739317+0000","last_undegraded":"2023-10-30T12:39:47.739317+0000","last_fullsized":"2023-10-30T12:39:47.739317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:17:53.592563+0000","last_clean_scrub_stamp":"2023-10-30T12:17:53.592563+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":849,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.512109+0000","last_change":"2023-10-30T12:19:25.736207+0000","last_active":"2023-10-30T12:39:43.512109+0000","last_peered":"2023-10-30T12:39:43.512109+0000","last_clean":"2023-10-30T12:39:43.512109+0000","last_became_active":"2023-10-30T12:19:25.715885+0000","last_became_peered":"2023-10-30T12:19:25.715885+0000","last_unstale":"2023-10-30T12:39:43.512109+0000","last_undegraded":"2023-10-30T12:39:43.512109+0000","last_fullsized":"2023-10-30T12:39:43.512109+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:50.517927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":843,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.466127+0000","last_change":"2023-10-30T12:19:29.928614+0000","last_active":"2023-10-30T12:39:43.466127+0000","last_peered":"2023-10-30T12:39:43.466127+0000","last_clean":"2023-10-30T12:39:43.466127+0000","last_became_active":"2023-10-30T12:19:29.879226+0000","last_became_peered":"2023-10-30T12:19:29.879226+0000","last_unstale":"2023-10-30T12:39:43.466127+0000","last_undegraded":"2023-10-30T12:39:43.466127+0000","last_fullsized":"2023-10-30T12:39:43.466127+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:26:19.751528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":847,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.489479+0000","last_change":"2023-10-30T12:19:40.799718+0000","last_active":"2023-10-30T12:39:43.489479+0000","last_peered":"2023-10-30T12:39:43.489479+0000","last_clean":"2023-10-30T12:39:43.489479+0000","last_became_active":"2023-10-30T12:19:40.794341+0000","last_became_peered":"2023-10-30T12:19:40.794341+0000","last_unstale":"2023-10-30T12:39:43.489479+0000","last_undegraded":"2023-10-30T12:39:43.489479+0000","last_fullsized":"2023-10-30T12:39:43.489479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:58.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":848,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.509587+0000","last_change":"2023-10-30T12:19:28.247323+0000","last_active":"2023-10-30T12:39:43.509587+0000","last_peered":"2023-10-30T12:39:43.509587+0000","last_clean":"2023-10-30T12:39:43.509587+0000","last_became_active":"2023-10-30T12:19:28.238513+0000","last_became_peered":"2023-10-30T12:19:28.238513+0000","last_unstale":"2023-10-30T12:39:43.509587+0000","last_undegraded":"2023-10-30T12:39:43.509587+0000","last_fullsized":"2023-10-30T12:39:43.509587+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:43.483993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":837,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:44.347307+0000","last_change":"2023-10-30T12:19:27.303856+0000","last_active":"2023-10-30T12:39:44.347307+0000","last_peered":"2023-10-30T12:39:44.347307+0000","last_clean":"2023-10-30T12:39:44.347307+0000","last_became_active":"2023-10-30T12:19:27.284772+0000","last_became_peered":"2023-10-30T12:19:27.284772+0000","last_unstale":"2023-10-30T12:39:44.347307+0000","last_undegraded":"2023-10-30T12:39:44.347307+0000","last_fullsized":"2023-10-30T12:39:44.347307+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:40.340431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":830,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:47.745638+0000","last_change":"2023-10-30T12:19:33.492307+0000","last_active":"2023-10-30T12:39:47.745638+0000","last_peered":"2023-10-30T12:39:47.745638+0000","last_clean":"2023-10-30T12:39:47.745638+0000","last_became_active":"2023-10-30T12:19:33.483122+0000","last_became_peered":"2023-10-30T12:19:33.483122+0000","last_unstale":"2023-10-30T12:39:47.745638+0000","last_undegraded":"2023-10-30T12:39:47.745638+0000","last_fullsized":"2023-10-30T12:39:47.745638+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":844,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.454972+0000","last_change":"2023-10-30T12:19:26.678316+0000","last_active":"2023-10-30T12:39:43.454972+0000","last_peered":"2023-10-30T12:39:43.454972+0000","last_clean":"2023-10-30T12:39:43.454972+0000","last_became_active":"2023-10-30T12:19:26.663005+0000","last_became_peered":"2023-10-30T12:19:26.663005+0000","last_unstale":"2023-10-30T12:39:43.454972+0000","last_undegraded":"2023-10-30T12:39:43.454972+0000","last_fullsized":"2023-10-30T12:39:43.454972+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:22:45.623416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":848,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.471419+0000","last_change":"2023-10-30T12:19:26.773408+0000","last_active":"2023-10-30T12:39:43.471419+0000","last_peered":"2023-10-30T12:39:43.471419+0000","last_clean":"2023-10-30T12:39:43.471419+0000","last_became_active":"2023-10-30T12:19:26.766234+0000","last_became_peered":"2023-10-30T12:19:26.766234+0000","last_unstale":"2023-10-30T12:39:43.471419+0000","last_undegraded":"2023-10-30T12:39:43.471419+0000","last_fullsized":"2023-10-30T12:39:43.471419+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:39.619411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":837,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.424792+0000","last_change":"2023-10-30T12:19:29.219323+0000","last_active":"2023-10-30T12:39:43.424792+0000","last_peered":"2023-10-30T12:39:43.424792+0000","last_clean":"2023-10-30T12:39:43.424792+0000","last_became_active":"2023-10-30T12:19:29.191499+0000","last_became_peered":"2023-10-30T12:19:29.191499+0000","last_unstale":"2023-10-30T12:39:43.424792+0000","last_undegraded":"2023-10-30T12:39:43.424792+0000","last_fullsized":"2023-10-30T12:39:43.424792+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:55.259618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":850,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:44.674758+0000","last_change":"2023-10-30T12:19:30.420980+0000","last_active":"2023-10-30T12:39:44.674758+0000","last_peered":"2023-10-30T12:39:44.674758+0000","last_clean":"2023-10-30T12:39:44.674758+0000","last_became_active":"2023-10-30T12:19:30.408141+0000","last_became_peered":"2023-10-30T12:19:30.408141+0000","last_unstale":"2023-10-30T12:39:44.674758+0000","last_undegraded":"2023-10-30T12:39:44.674758+0000","last_fullsized":"2023-10-30T12:39:44.674758+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:04:46.816451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":837,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.852214+0000","last_change":"2023-10-30T12:19:40.539149+0000","last_active":"2023-10-30T12:39:43.852214+0000","last_peered":"2023-10-30T12:39:43.852214+0000","last_clean":"2023-10-30T12:39:43.852214+0000","last_became_active":"2023-10-30T12:19:40.528642+0000","last_became_peered":"2023-10-30T12:19:40.528642+0000","last_unstale":"2023-10-30T12:39:43.852214+0000","last_undegraded":"2023-10-30T12:39:43.852214+0000","last_fullsized":"2023-10-30T12:39:43.852214+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:34.653320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":838,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:39:43.836022+0000","last_change":"2023-10-30T12:19:30.748646+0000","last_active":"2023-10-30T12:39:43.836022+0000","last_peered":"2023-10-30T12:39:43.836022+0000","last_clean":"2023-10-30T12:39:43.836022+0000","last_became_active":"2023-10-30T12:19:30.720900+0000","last_became_peered":"2023-10-30T12:19:30.720900+0000","last_unstale":"2023-10-30T12:39:43.836022+0000","last_undegraded":"2023-10-30T12:39:43.836022+0000","last_fullsized":"2023-10-30T12:39:43.836022+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:59.574207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509707,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62340,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309500,"statfs":{"total":96636764160,"available":96572928000,"internally_reserved":0,"allocated":8970240,"data_stored":4460959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":23.231999999999999,"5min":17.655999999999999,"15min":16.648},"min":{"1min":6.4530000000000003,"5min":4.0250000000000004,"15min":4.0250000000000004},"max":{"1min":181.905,"5min":233.52500000000001,"15min":233.52500000000001},"last":7.476},{"interface":"front","average":{"1min":27.143000000000001,"5min":21.59,"15min":22.366},"min":{"1min":9.7650000000000006,"5min":5.6740000000000004,"15min":5.6740000000000004},"max":{"1min":183.072,"5min":214.31899999999999,"15min":365.47399999999999},"last":13.722}]},{"osd":1,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.035,"5min":12.183,"15min":14.696},"min":{"1min":6.0800000000000001,"5min":4.4359999999999999,"15min":4.2389999999999999},"max":{"1min":63.286999999999999,"5min":63.286999999999999,"15min":142.51400000000001},"last":8.6660000000000004},{"interface":"front","average":{"1min":17.978999999999999,"5min":14.823,"15min":17.242000000000001},"min":{"1min":6.056,"5min":5.1689999999999996,"15min":5.1669999999999998},"max":{"1min":61.552,"5min":112.276,"15min":211.11600000000001},"last":7.1239999999999997}]},{"osd":2,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.715,"5min":21.622,"15min":20.561},"min":{"1min":6.5259999999999998,"5min":6.2530000000000001,"15min":6.2530000000000001},"max":{"1min":61.927,"5min":187.00399999999999,"15min":187.00399999999999},"last":12.048},{"interface":"front","average":{"1min":20.795999999999999,"5min":23.681999999999999,"15min":21.565000000000001},"min":{"1min":8.2710000000000008,"5min":6.5359999999999996,"15min":5.6260000000000003},"max":{"1min":80.686999999999998,"5min":240.99799999999999,"15min":240.99799999999999},"last":9.9830000000000005}]},{"osd":3,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":20.48,"5min":26.420000000000002,"15min":25.207999999999998},"min":{"1min":10.49,"5min":7.5990000000000002,"15min":6.2240000000000002},"max":{"1min":62.517000000000003,"5min":279.94099999999997,"15min":279.94099999999997},"last":15.25},{"interface":"front","average":{"1min":17.039000000000001,"5min":24.077999999999999,"15min":23.585000000000001},"min":{"1min":8.1370000000000005,"5min":6.617,"15min":6.5960000000000001},"max":{"1min":55.351999999999997,"5min":180.74600000000001,"15min":180.74600000000001},"last":14.371}]},{"osd":4,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":13.237,"5min":19.042000000000002,"15min":18.274999999999999},"min":{"1min":6.617,"5min":6.617,"15min":6.1840000000000002},"max":{"1min":59.103000000000002,"5min":184.88,"15min":184.88},"last":9.3000000000000007},{"interface":"front","average":{"1min":20.768999999999998,"5min":26.350000000000001,"15min":25.687999999999999},"min":{"1min":11.442,"5min":9.4160000000000004,"15min":7.3540000000000001},"max":{"1min":54.426000000000002,"5min":374.38900000000001,"15min":374.38900000000001},"last":15.821999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":19.882000000000001,"5min":25.530999999999999,"15min":25.718},"min":{"1min":10.507,"5min":8.1210000000000004,"15min":6.2729999999999997},"max":{"1min":43.252000000000002,"5min":218.28800000000001,"15min":218.28800000000001},"last":12.664999999999999},{"interface":"front","average":{"1min":17.363,"5min":25.651,"15min":23.794},"min":{"1min":6.7160000000000002,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":34.281999999999996,"5min":195.74700000000001,"15min":195.74700000000001},"last":11.382}]},{"osd":6,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":21.611999999999998,"5min":23.324000000000002,"15min":27.591999999999999},"min":{"1min":9.8620000000000001,"5min":9.593,"15min":8.3320000000000007},"max":{"1min":69.704999999999998,"5min":178.68100000000001,"15min":224.71000000000001},"last":17.207000000000001},{"interface":"front","average":{"1min":19.506,"5min":24.533000000000001,"15min":25.074000000000002},"min":{"1min":9.7799999999999994,"5min":7.2469999999999999,"15min":7.2469999999999999},"max":{"1min":39.325000000000003,"5min":187.14500000000001,"15min":210.05199999999999},"last":16.585999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80976,"kb_used_data":9348,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94290864,"statfs":{"total":96636764160,"available":96553844736,"internally_reserved":0,"allocated":9572352,"data_stored":4982006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:46 2023","interfaces":[{"interface":"back","average":{"1min":13.422000000000001,"5min":16.344999999999999,"15min":23.686},"min":{"1min":4.9379999999999997,"5min":4.1050000000000004,"15min":4.1050000000000004},"max":{"1min":61.497999999999998,"5min":158.87799999999999,"15min":308.51999999999998},"last":6.601},{"interface":"front","average":{"1min":15.584,"5min":17.266999999999999,"15min":22.806999999999999},"min":{"1min":5.7670000000000003,"5min":4.7469999999999999,"15min":4.7469999999999999},"max":{"1min":66.950999999999993,"5min":146.374,"15min":244.072},"last":7.8419999999999996}]},{"osd":1,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":18.443999999999999,"5min":16.146000000000001,"15min":28.567},"min":{"1min":5.8040000000000003,"5min":5.5300000000000002,"15min":5.5300000000000002},"max":{"1min":138.964,"5min":138.964,"15min":620.21500000000003},"last":15.868},{"interface":"front","average":{"1min":17.212,"5min":17.853000000000002,"15min":27.408000000000001},"min":{"1min":7.1970000000000001,"5min":5.6150000000000002,"15min":5.4109999999999996},"max":{"1min":93.744,"5min":162.25200000000001,"15min":334.62},"last":8.875}]},{"osd":2,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":18.535,"5min":18.928000000000001,"15min":23.741},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":134.77000000000001,"5min":208.28100000000001,"15min":246.37},"last":10.766},{"interface":"front","average":{"1min":21.039000000000001,"5min":20.257999999999999,"15min":24.73},"min":{"1min":8.4540000000000006,"5min":6.3049999999999997,"15min":5.2770000000000001},"max":{"1min":141.99799999999999,"5min":141.99799999999999,"15min":348.41000000000003},"last":9.5340000000000007}]},{"osd":3,"last update":"Mon Oct 30 12:39:26 2023","interfaces":[{"interface":"back","average":{"1min":19.34,"5min":23.516999999999999,"15min":25.564},"min":{"1min":6.8339999999999996,"5min":6.3860000000000001,"15min":6.3860000000000001},"max":{"1min":97.822999999999993,"5min":148.125,"15min":273.10300000000001},"last":16.527999999999999},{"interface":"front","average":{"1min":24.561,"5min":21.800000000000001,"15min":24.858000000000001},"min":{"1min":9.3970000000000002,"5min":6.2889999999999997,"15min":5.8029999999999999},"max":{"1min":94.296000000000006,"5min":154.857,"15min":282.28500000000003},"last":13.036}]},{"osd":4,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":25.477,"5min":23.068000000000001,"15min":33.439},"min":{"1min":11.792999999999999,"5min":9.5109999999999992,"15min":7.0259999999999998},"max":{"1min":144.102,"5min":144.102,"15min":512.01199999999994},"last":13.657},{"interface":"front","average":{"1min":27.440999999999999,"5min":25.501000000000001,"15min":31.102},"min":{"1min":7.4530000000000003,"5min":7.4530000000000003,"15min":6.3890000000000002},"max":{"1min":113.208,"5min":169.45500000000001,"15min":386.62200000000001},"last":12.337}]},{"osd":6,"last update":"Mon Oct 30 12:38:52 2023","interfaces":[{"interface":"back","average":{"1min":28.742000000000001,"5min":28.922000000000001,"15min":31.957000000000001},"min":{"1min":14.225,"5min":11.138,"15min":7.9729999999999999},"max":{"1min":144.726,"5min":242.16800000000001,"15min":588.93899999999996},"last":14.975},{"interface":"front","average":{"1min":32.106999999999999,"5min":34.963999999999999,"15min":37.119},"min":{"1min":16.084,"5min":12.661,"15min":7.25},"max":{"1min":161.18000000000001,"5min":319.17700000000002,"15min":500.173},"last":18.384}]},{"osd":7,"last update":"Mon Oct 30 12:39:26 2023","interfaces":[{"interface":"back","average":{"1min":20.66,"5min":27.561,"15min":28.265999999999998},"min":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"max":{"1min":113.35899999999999,"5min":258.61200000000002,"15min":258.61200000000002},"last":11.766999999999999},{"interface":"front","average":{"1min":16.838999999999999,"5min":22.385999999999999,"15min":24.34},"min":{"1min":6.109,"5min":6.109,"15min":6.109},"max":{"1min":55.759,"5min":145.15000000000001,"15min":245.71899999999999},"last":17.568000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509707,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62340,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309500,"statfs":{"total":96636764160,"available":96572928000,"internally_reserved":0,"allocated":8970240,"data_stored":4460959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":8.8330000000000002,"5min":15.872,"15min":22.242999999999999},"min":{"1min":4.7569999999999997,"5min":4.2140000000000004,"15min":4.0620000000000003},"max":{"1min":16.516999999999999,"5min":158.499,"15min":295.56400000000002},"last":5.4720000000000004},{"interface":"front","average":{"1min":8.3000000000000007,"5min":18.66,"15min":23.565999999999999},"min":{"1min":4.4080000000000004,"5min":4.4080000000000004,"15min":4.4080000000000004},"max":{"1min":14.081,"5min":182.96600000000001,"15min":368.01900000000001},"last":6.827}]},{"osd":1,"last update":"Mon Oct 30 12:39:46 2023","interfaces":[{"interface":"back","average":{"1min":11.259,"5min":16.731000000000002,"15min":20.905999999999999},"min":{"1min":6.633,"5min":5.6420000000000003,"15min":4.5389999999999997},"max":{"1min":18.992999999999999,"5min":171.24100000000001,"15min":322.97800000000001},"last":12.804},{"interface":"front","average":{"1min":15.079000000000001,"5min":21.649000000000001,"15min":20.692},"min":{"1min":6.6539999999999999,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":32.031999999999996,"5min":168.833,"15min":245.44900000000001},"last":8.5760000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:38:51 2023","interfaces":[{"interface":"back","average":{"1min":18.649999999999999,"5min":26.581,"15min":25.079000000000001},"min":{"1min":7.9740000000000002,"5min":6.0039999999999996,"15min":6.0039999999999996},"max":{"1min":112.88,"5min":316.74799999999999,"15min":316.74799999999999},"last":13.507},{"interface":"front","average":{"1min":16.074999999999999,"5min":26.199999999999999,"15min":26.724},"min":{"1min":8.0690000000000008,"5min":5.9489999999999998,"15min":5.9489999999999998},"max":{"1min":53.756,"5min":317.55399999999997,"15min":317.55399999999997},"last":14.148}]},{"osd":3,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":14.321999999999999,"5min":23.728999999999999,"15min":28.404},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":6.7270000000000003},"max":{"1min":21.91,"5min":161.197,"15min":383.87099999999998},"last":17.016999999999999},{"interface":"front","average":{"1min":13.757999999999999,"5min":26.026,"15min":26.616},"min":{"1min":8.5,"5min":8.5,"15min":6.6529999999999996},"max":{"1min":18.109999999999999,"5min":260.94200000000001,"15min":284.61700000000002},"last":14.864000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:39:46 2023","interfaces":[{"interface":"back","average":{"1min":19.042999999999999,"5min":27.260999999999999,"15min":27.414000000000001},"min":{"1min":8.2910000000000004,"5min":7.9429999999999996,"15min":5.7119999999999997},"max":{"1min":51.045999999999999,"5min":185.47200000000001,"15min":261.80200000000002},"last":16.192},{"interface":"front","average":{"1min":16.626999999999999,"5min":21.643000000000001,"15min":26.568000000000001},"min":{"1min":8.0180000000000007,"5min":6.3760000000000003,"15min":6.3760000000000003},"max":{"1min":33.390999999999998,"5min":174.38999999999999,"15min":371.35700000000003},"last":9.8379999999999992}]},{"osd":6,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":23.914999999999999,"5min":27.789999999999999,"15min":27.952999999999999},"min":{"1min":10.460000000000001,"5min":9.4939999999999998,"15min":8.0060000000000002},"max":{"1min":95.795000000000002,"5min":215.31800000000001,"15min":312.44999999999999},"last":18.387},{"interface":"front","average":{"1min":19.725999999999999,"5min":28.399999999999999,"15min":27.664000000000001},"min":{"1min":9.859,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":63.609999999999999,"5min":266.50599999999997,"15min":320.50999999999999},"last":17.654}]},{"osd":7,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":17.331,"5min":26.408000000000001,"15min":28.349},"min":{"1min":9.048,"5min":9.048,"15min":7.9370000000000003},"max":{"1min":55.015000000000001,"5min":193.95400000000001,"15min":319.84199999999998},"last":11.736000000000001},{"interface":"front","average":{"1min":15.836,"5min":26.617000000000001,"15min":27.077000000000002},"min":{"1min":10.515000000000001,"5min":8.9489999999999998,"15min":8.5410000000000004},"max":{"1min":28.527000000000001,"5min":252.232,"15min":315.06900000000002},"last":19.75}]}]},{"osd":3,"up_from":15,"seq":64424509705,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84548,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287292,"statfs":{"total":96636764160,"available":96550187008,"internally_reserved":0,"allocated":8970240,"data_stored":4460959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.731999999999999,"5min":21.190000000000001,"15min":21.754999999999999},"min":{"1min":4.609,"5min":4.2460000000000004,"15min":4.1159999999999997},"max":{"1min":210.398,"5min":303.58600000000001,"15min":303.58600000000001},"last":8.4619999999999997},{"interface":"front","average":{"1min":26.295999999999999,"5min":17.559999999999999,"15min":21.213000000000001},"min":{"1min":6.0049999999999999,"5min":4.8819999999999997,"15min":4.1070000000000002},"max":{"1min":187.82400000000001,"5min":187.82400000000001,"15min":268.68799999999999},"last":10.436}]},{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":42.889000000000003,"5min":25.684999999999999,"15min":23.533999999999999},"min":{"1min":7.7990000000000004,"5min":5.6070000000000002,"15min":4.8940000000000001},"max":{"1min":220.96600000000001,"5min":220.96600000000001,"15min":389.303},"last":7.6470000000000002},{"interface":"front","average":{"1min":43.094999999999999,"5min":25.721,"15min":24.445},"min":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":5.0890000000000004},"max":{"1min":222.328,"5min":222.328,"15min":468.13299999999998},"last":11.372999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":19.759,"5min":21.931000000000001,"15min":22.350999999999999},"min":{"1min":8.0250000000000004,"5min":6.8230000000000004,"15min":5.6150000000000002},"max":{"1min":135.97399999999999,"5min":352.09399999999999,"15min":352.09399999999999},"last":12.135999999999999},{"interface":"front","average":{"1min":18.385000000000002,"5min":20.425999999999998,"15min":23.646999999999998},"min":{"1min":7.7889999999999997,"5min":6.9850000000000003,"15min":6.9850000000000003},"max":{"1min":94.552999999999997,"5min":184.04599999999999,"15min":287.33699999999999},"last":13.625}]},{"osd":4,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.673000000000002,"5min":25.640000000000001,"15min":28.405000000000001},"min":{"1min":8.6929999999999996,"5min":7.1799999999999997,"15min":6.5010000000000003},"max":{"1min":211.03200000000001,"5min":310.12099999999998,"15min":310.12099999999998},"last":16.123000000000001},{"interface":"front","average":{"1min":31.611000000000001,"5min":25.715,"15min":25.268000000000001},"min":{"1min":9.2690000000000001,"5min":6.7169999999999996,"15min":6.0919999999999996},"max":{"1min":187.15700000000001,"5min":359.87799999999999,"15min":359.87799999999999},"last":9.4390000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":44.430999999999997,"5min":29.558,"15min":30.042999999999999},"min":{"1min":9.8840000000000003,"5min":9,"15min":6.851},"max":{"1min":274.15300000000002,"5min":360.88099999999997,"15min":564.26700000000005},"last":12.98},{"interface":"front","average":{"1min":50.723999999999997,"5min":28.849,"15min":28.324999999999999},"min":{"1min":10.619,"5min":8.8680000000000003,"15min":8.0109999999999992},"max":{"1min":289.68099999999998,"5min":289.68099999999998,"15min":294.392},"last":18.986000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":30.876999999999999,"5min":29.062999999999999,"15min":30.353999999999999},"min":{"1min":11.016,"5min":9.3369999999999997,"15min":8.8149999999999995},"max":{"1min":197.09200000000001,"5min":374.33999999999997,"15min":374.33999999999997},"last":14.358000000000001},{"interface":"front","average":{"1min":33.301000000000002,"5min":31.440000000000001,"15min":31.552},"min":{"1min":11.416,"5min":11.416,"15min":8.9079999999999995},"max":{"1min":177.63499999999999,"5min":352.79500000000002,"15min":352.79500000000002},"last":16.827000000000002}]},{"osd":7,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":36.420000000000002,"5min":32.140999999999998,"15min":34.933},"min":{"1min":12.868,"5min":11.401999999999999,"15min":8.5099999999999998},"max":{"1min":203.70099999999999,"5min":382.68299999999999,"15min":382.68299999999999},"last":22.288},{"interface":"front","average":{"1min":24.998999999999999,"5min":29.645,"15min":32.134},"min":{"1min":11.976000000000001,"5min":9.0850000000000009,"15min":8.7699999999999996},"max":{"1min":86.024000000000001,"5min":361.47199999999998,"15min":562.05700000000002},"last":17.765999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509705,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66116,"kb_used_data":8376,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94305724,"statfs":{"total":96636764160,"available":96569061376,"internally_reserved":0,"allocated":8577024,"data_stored":4196328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:02 2023","interfaces":[{"interface":"back","average":{"1min":9.0120000000000005,"5min":15.948,"15min":19.835000000000001},"min":{"1min":4.4260000000000002,"5min":4.4260000000000002,"15min":3.8079999999999998},"max":{"1min":20.417000000000002,"5min":121.476,"15min":337.86700000000002},"last":5.2050000000000001},{"interface":"front","average":{"1min":10.103,"5min":14.271000000000001,"15min":18.513000000000002},"min":{"1min":5.6310000000000002,"5min":4.4279999999999999,"15min":4.1399999999999997},"max":{"1min":18.503,"5min":119.87,"15min":343.983},"last":12.718}]},{"osd":1,"last update":"Mon Oct 30 12:38:53 2023","interfaces":[{"interface":"back","average":{"1min":10.884,"5min":16.09,"15min":23.937000000000001},"min":{"1min":7.0860000000000003,"5min":4.9790000000000001,"15min":4.6559999999999997},"max":{"1min":16.172000000000001,"5min":153.751,"15min":501.19900000000001},"last":14.792},{"interface":"front","average":{"1min":13.513999999999999,"5min":18.212,"15min":22.693999999999999},"min":{"1min":6.9859999999999998,"5min":6.9859999999999998,"15min":6.649},"max":{"1min":34.274000000000001,"5min":122.767,"15min":391.58300000000003},"last":8.2680000000000007}]},{"osd":3,"last update":"Mon Oct 30 12:38:57 2023","interfaces":[{"interface":"back","average":{"1min":13.710000000000001,"5min":21.341999999999999,"15min":27.303000000000001},"min":{"1min":7.7140000000000004,"5min":6.923,"15min":6.6379999999999999},"max":{"1min":34.911999999999999,"5min":173.56899999999999,"15min":414.69299999999998},"last":10.462999999999999},{"interface":"front","average":{"1min":14.257,"5min":21.814,"15min":28.373000000000001},"min":{"1min":7.484,"5min":7.1769999999999996,"15min":6.6269999999999998},"max":{"1min":32.726999999999997,"5min":197.904,"15min":502.19299999999998},"last":16.469999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:39:42 2023","interfaces":[{"interface":"back","average":{"1min":16.821999999999999,"5min":22.812000000000001,"15min":23.850999999999999},"min":{"1min":8,"5min":8,"15min":6.415},"max":{"1min":40.204999999999998,"5min":157.77199999999999,"15min":331.834},"last":15.547000000000001},{"interface":"front","average":{"1min":15.441000000000001,"5min":17.044,"15min":18.234999999999999},"min":{"1min":6.8239999999999998,"5min":6.1509999999999998,"15min":5.984},"max":{"1min":47.32,"5min":134.452,"15min":179.89099999999999},"last":9.3780000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:39:47 2023","interfaces":[{"interface":"back","average":{"1min":19.218,"5min":22.625,"15min":28.466999999999999},"min":{"1min":9.5459999999999994,"5min":8.0619999999999994,"15min":7.4500000000000002},"max":{"1min":42.088000000000001,"5min":201.00299999999999,"15min":326.47699999999998},"last":17.251999999999999},{"interface":"front","average":{"1min":18.844999999999999,"5min":24.981000000000002,"15min":30.864999999999998},"min":{"1min":8.9949999999999992,"5min":8.9949999999999992,"15min":8.9949999999999992},"max":{"1min":40.993000000000002,"5min":238.57300000000001,"15min":338.82100000000003},"last":13.191000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":16.959,"5min":19.983000000000001,"15min":20.140000000000001},"min":{"1min":8.1110000000000007,"5min":6.3239999999999998,"15min":6.1100000000000003},"max":{"1min":49.598999999999997,"5min":146.75,"15min":182.97499999999999},"last":12.234999999999999},{"interface":"front","average":{"1min":20.119,"5min":23.864000000000001,"15min":25.939},"min":{"1min":11.311,"5min":9.8819999999999997,"15min":8.7729999999999997},"max":{"1min":45.713000000000001,"5min":154.619,"15min":315.71100000000001},"last":18.016999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:38:53 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":18.695,"15min":22.042999999999999},"min":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"max":{"1min":27.41,"5min":123.926,"15min":228.67400000000001},"last":13.738},{"interface":"front","average":{"1min":19.202000000000002,"5min":25.521000000000001,"15min":29.041},"min":{"1min":11.582000000000001,"5min":10.462999999999999,"15min":9.0589999999999993},"max":{"1min":38.158000000000001,"5min":160.93600000000001,"15min":328.51799999999997},"last":18.972000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509706,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62924,"kb_used_data":9344,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94308916,"statfs":{"total":96636764160,"available":96572329984,"internally_reserved":0,"allocated":9568256,"data_stored":4982001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":10.207000000000001,"5min":11.294,"15min":18.631},"min":{"1min":5.5519999999999996,"5min":4.3760000000000003,"15min":4.3760000000000003},"max":{"1min":24.184999999999999,"5min":75.006,"15min":294.67099999999999},"last":10.103},{"interface":"front","average":{"1min":11.981999999999999,"5min":11.672000000000001,"15min":17.254000000000001},"min":{"1min":6.4210000000000003,"5min":5.2400000000000002,"15min":4.952},"max":{"1min":27.530999999999999,"5min":61.932000000000002,"15min":241.066},"last":6.8810000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":16.41,"5min":14.955,"15min":19.184000000000001},"min":{"1min":6.4299999999999997,"5min":6.4299999999999997,"15min":6.0519999999999996},"max":{"1min":67.140000000000001,"5min":75.491,"15min":295.50599999999997},"last":10.787000000000001},{"interface":"front","average":{"1min":18.949999999999999,"5min":13.51,"15min":19.122},"min":{"1min":6.8810000000000002,"5min":6.1580000000000004,"15min":5.327},"max":{"1min":135.00800000000001,"5min":135.00800000000001,"15min":269.99200000000002},"last":12.981}]},{"osd":3,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":19.390000000000001,"15min":21.367999999999999},"min":{"1min":6.9889999999999999,"5min":6.4779999999999998,"15min":6.4779999999999998},"max":{"1min":25.774999999999999,"5min":310.40800000000002,"15min":551.58000000000004},"last":13.760999999999999},{"interface":"front","average":{"1min":17.298999999999999,"5min":28.062000000000001,"15min":27.957000000000001},"min":{"1min":9.8819999999999997,"5min":9.8819999999999997,"15min":9.1999999999999993},"max":{"1min":29.579000000000001,"5min":338.88600000000002,"15min":562.06899999999996},"last":17.873000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":14.186,"5min":16.617000000000001,"15min":21.454000000000001},"min":{"1min":7.0469999999999997,"5min":6.7480000000000002,"15min":6.6109999999999998},"max":{"1min":40.465000000000003,"5min":84.212000000000003,"15min":285.26499999999999},"last":7.8540000000000001},{"interface":"front","average":{"1min":14.521000000000001,"5min":16.108000000000001,"15min":22.126000000000001},"min":{"1min":9.0500000000000007,"5min":6.7450000000000001,"15min":6.7450000000000001},"max":{"1min":42.087000000000003,"5min":62.616,"15min":327.59699999999998},"last":12.089}]},{"osd":5,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":35.115000000000002,"5min":25.553999999999998,"15min":26.443999999999999},"min":{"1min":9.9410000000000007,"5min":8.1639999999999997,"15min":7.3070000000000004},"max":{"1min":322.59899999999999,"5min":322.59899999999999,"15min":322.59899999999999},"last":15.997},{"interface":"front","average":{"1min":32.095999999999997,"5min":21.616,"15min":24.370999999999999},"min":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"max":{"1min":283.06,"5min":283.06,"15min":286.43799999999999},"last":17.096}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":13.723000000000001,"5min":17.039000000000001,"15min":22.716000000000001},"min":{"1min":7.6059999999999999,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":28.151,"5min":71.662999999999997,"15min":348.91000000000003},"last":9.2100000000000009},{"interface":"front","average":{"1min":18.815999999999999,"5min":23.582999999999998,"15min":26.449000000000002},"min":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":8.4819999999999993},"max":{"1min":33.466000000000001,"5min":99.231999999999999,"15min":301.99799999999999},"last":18.454000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":21.850999999999999,"5min":21.949000000000002,"15min":25.800999999999998},"min":{"1min":13.108000000000001,"5min":9.2780000000000005,"15min":7.8739999999999997},"max":{"1min":68.974999999999994,"5min":107.703,"15min":305.19900000000001},"last":19.542000000000002},{"interface":"front","average":{"1min":20.207000000000001,"5min":20.809000000000001,"15min":26.821000000000002},"min":{"1min":8.8640000000000008,"5min":8.8640000000000008,"15min":8.0920000000000005},"max":{"1min":70.406999999999996,"5min":106.678,"15min":277.05200000000002},"last":15.051}]}]},{"osd":0,"up_from":15,"seq":64424509706,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62344,"kb_used_data":8764,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309496,"statfs":{"total":96636764160,"available":96572923904,"internally_reserved":0,"allocated":8974336,"data_stored":4460964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":14.013,"5min":16.620999999999999,"15min":19.768000000000001},"min":{"1min":6.117,"5min":4.1799999999999997,"15min":4.1799999999999997},"max":{"1min":73.113,"5min":232.798,"15min":287.096},"last":9.9030000000000005},{"interface":"front","average":{"1min":11.015000000000001,"5min":14.513,"15min":22.515999999999998},"min":{"1min":7.6879999999999997,"5min":5.2370000000000001,"15min":4.7590000000000003},"max":{"1min":29.097000000000001,"5min":218.61099999999999,"15min":425.762},"last":6.702}]},{"osd":2,"last update":"Mon Oct 30 12:39:40 2023","interfaces":[{"interface":"back","average":{"1min":21.314,"5min":17.907,"15min":20.555},"min":{"1min":8.3599999999999994,"5min":5.2480000000000002,"15min":5.2480000000000002},"max":{"1min":78.947999999999993,"5min":228.90799999999999,"15min":299.46600000000001},"last":13.756},{"interface":"front","average":{"1min":28.640000000000001,"5min":18.164000000000001,"15min":20.786000000000001},"min":{"1min":6.7779999999999996,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":113.76000000000001,"5min":138.13499999999999,"15min":258.45400000000001},"last":16.472999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.376999999999999,"5min":16.640999999999998,"15min":20.597999999999999},"min":{"1min":9.6329999999999991,"5min":6.5499999999999998,"15min":5.2110000000000003},"max":{"1min":41.456000000000003,"5min":166.21600000000001,"15min":264.97199999999998},"last":11.222},{"interface":"front","average":{"1min":19.736000000000001,"5min":21.084,"15min":25.367000000000001},"min":{"1min":8.4079999999999995,"5min":8.4079999999999995,"15min":7.218},"max":{"1min":54.859000000000002,"5min":230.73699999999999,"15min":258.93200000000002},"last":14.382}]},{"osd":4,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6620000000000008,"5min":18.376000000000001,"15min":22.655999999999999},"min":{"1min":5.681,"5min":5.681,"15min":5.681},"max":{"1min":24.405999999999999,"5min":209.64599999999999,"15min":209.64599999999999},"last":8.2089999999999996},{"interface":"front","average":{"1min":20.053999999999998,"5min":25.309000000000001,"15min":23.466999999999999},"min":{"1min":8.8819999999999997,"5min":8.0990000000000002,"15min":5.8499999999999996},"max":{"1min":90.787999999999997,"5min":208.92400000000001,"15min":208.92400000000001},"last":19.209}]},{"osd":5,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.98,"5min":29.106000000000002,"15min":29.109999999999999},"min":{"1min":10.965,"5min":8.9440000000000008,"15min":7.5510000000000002},"max":{"1min":31.939,"5min":245.16300000000001,"15min":285.57100000000003},"last":17.798999999999999},{"interface":"front","average":{"1min":17.673999999999999,"5min":21.956,"15min":28.129999999999999},"min":{"1min":10.518000000000001,"5min":7.2720000000000002,"15min":7.2720000000000002},"max":{"1min":30.361000000000001,"5min":142.25,"15min":292.07900000000001},"last":17.132000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":15.827999999999999,"5min":20.135999999999999,"15min":21.259},"min":{"1min":7.7000000000000002,"5min":7.4370000000000003,"15min":7.4370000000000003},"max":{"1min":43.283999999999999,"5min":171.99600000000001,"15min":236.40000000000001},"last":9.3070000000000004},{"interface":"front","average":{"1min":18.968,"5min":24.116,"15min":25.454999999999998},"min":{"1min":12.252000000000001,"5min":10.470000000000001,"15min":8.9199999999999999},"max":{"1min":31.385000000000002,"5min":178.59999999999999,"15min":220.68700000000001},"last":11.83}]},{"osd":7,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":19.300999999999998,"5min":20.318000000000001,"15min":24.036000000000001},"min":{"1min":11.853,"5min":9.2100000000000009,"15min":9.2100000000000009},"max":{"1min":32.664999999999999,"5min":143.49600000000001,"15min":245.87100000000001},"last":16.067},{"interface":"front","average":{"1min":16.231000000000002,"5min":22.702000000000002,"15min":24.879999999999999},"min":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":32.033000000000001,"5min":181.15100000000001,"15min":260.04000000000002},"last":12.526999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575114,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61956,"kb_used_data":8376,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309884,"statfs":{"total":96636764160,"available":96573321216,"internally_reserved":0,"allocated":8577024,"data_stored":4196328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":14.144,"5min":11.411,"15min":20.879000000000001},"min":{"1min":4.5,"5min":4.0960000000000001,"15min":4.0099999999999998},"max":{"1min":110.503,"5min":110.503,"15min":623.43700000000001},"last":10.747999999999999},{"interface":"front","average":{"1min":10.115,"5min":11.086,"15min":18.956},"min":{"1min":4.9050000000000002,"5min":4.202,"15min":4.202},"max":{"1min":36.860999999999997,"5min":51.253999999999998,"15min":563.08900000000006},"last":10.048999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":16.497,"5min":13.163,"15min":16.117000000000001},"min":{"1min":6.3159999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"max":{"1min":143.386,"5min":143.386,"15min":159.941},"last":5.431},{"interface":"front","average":{"1min":21.492000000000001,"5min":17.748000000000001,"15min":21.466000000000001},"min":{"1min":9.5489999999999995,"5min":7.0679999999999996,"15min":7.008},"max":{"1min":153.08199999999999,"5min":153.08199999999999,"15min":266.30599999999998},"last":9.6110000000000007}]},{"osd":2,"last update":"Mon Oct 30 12:38:51 2023","interfaces":[{"interface":"back","average":{"1min":11.6,"5min":18.939,"15min":22.053999999999998},"min":{"1min":6.1749999999999998,"5min":6.1749999999999998,"15min":5.0529999999999999},"max":{"1min":20.262,"5min":329.81999999999999,"15min":329.81999999999999},"last":12.071},{"interface":"front","average":{"1min":13.555999999999999,"5min":17.236000000000001,"15min":22.600000000000001},"min":{"1min":7.5229999999999997,"5min":6.8129999999999997,"15min":6.0350000000000001},"max":{"1min":18.695,"5min":132.25800000000001,"15min":245.63},"last":21.193999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":20.202999999999999,"5min":19.655999999999999,"15min":23.643000000000001},"min":{"1min":8.0839999999999996,"5min":6.4950000000000001,"15min":6.4950000000000001},"max":{"1min":124.31399999999999,"5min":124.31399999999999,"15min":185.98400000000001},"last":13.914999999999999},{"interface":"front","average":{"1min":22.238,"5min":22.606999999999999,"15min":24.629999999999999},"min":{"1min":8.4100000000000001,"5min":7.7000000000000002,"15min":6.8159999999999998},"max":{"1min":159.69999999999999,"5min":247.65899999999999,"15min":247.65899999999999},"last":14.917}]},{"osd":4,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":17.640000000000001,"5min":19.411000000000001,"15min":24.236999999999998},"min":{"1min":9.4979999999999993,"5min":5.8150000000000004,"15min":5.8150000000000004},"max":{"1min":50.786000000000001,"5min":117.967,"15min":232.17099999999999},"last":22.285},{"interface":"front","average":{"1min":18.247,"5min":18.09,"15min":22.715},"min":{"1min":6.806,"5min":6.806,"15min":6.6280000000000001},"max":{"1min":100.952,"5min":100.952,"15min":267.11000000000001},"last":15.616}]},{"osd":5,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":20.390000000000001,"5min":22.111000000000001,"15min":27.120999999999999},"min":{"1min":7.6349999999999998,"5min":7.1440000000000001,"15min":6.5110000000000001},"max":{"1min":125.926,"5min":186.54499999999999,"15min":203.66399999999999},"last":16.443999999999999},{"interface":"front","average":{"1min":16.417999999999999,"5min":23.021999999999998,"15min":27.204000000000001},"min":{"1min":8.6980000000000004,"5min":8.6980000000000004,"15min":7.1500000000000004},"max":{"1min":46.524000000000001,"5min":185.61099999999999,"15min":275.88099999999997},"last":12.746}]},{"osd":7,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":18.094000000000001,"5min":20.02,"15min":26.045000000000002},"min":{"1min":8.952,"5min":8.952,"15min":7.5579999999999998},"max":{"1min":50.177999999999997,"5min":99.480999999999995,"15min":227.18600000000001},"last":23.327000000000002},{"interface":"front","average":{"1min":17.981000000000002,"5min":21.117000000000001,"15min":25.186},"min":{"1min":9.5609999999999999,"5min":8.8019999999999996,"15min":8.8019999999999996},"max":{"1min":51.954000000000001,"5min":108.705,"15min":264.99299999999999},"last":17.917999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:39:55.013 INFO:tasks.ceph:pgid 20.8 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.013 INFO:tasks.ceph:pgid 20.9 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.014 INFO:tasks.ceph:pgid 20.a last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.014 INFO:tasks.ceph:pgid 20.b last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.014 INFO:tasks.ceph:pgid 20.c last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.015 INFO:tasks.ceph:pgid 20.d last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.015 INFO:tasks.ceph:pgid 20.e last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.015 INFO:tasks.ceph:pgid 20.f last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.015 INFO:tasks.ceph:pgid 20.0 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.016 INFO:tasks.ceph:pgid 20.1 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.016 INFO:tasks.ceph:pgid 20.2 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.016 INFO:tasks.ceph:pgid 20.3 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.017 INFO:tasks.ceph:pgid 20.11 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.017 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T12:17:59.043368+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=17, tm_sec=59, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.017 INFO:tasks.ceph:pgid 20.10 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.017 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T12:17:59.043368+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=17, tm_sec=59, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.018 INFO:tasks.ceph:pgid 20.13 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.018 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T12:17:59.043368+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=17, tm_sec=59, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.018 INFO:tasks.ceph:pgid 20.12 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.019 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-30T12:17:59.043368+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=17, tm_sec=59, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.019 INFO:tasks.ceph:pgid 20.14 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.019 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T12:17:59.043368+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=17, tm_sec=59, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.019 INFO:tasks.ceph:pgid 20.17 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.020 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-30T12:17:59.043368+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=17, tm_sec=59, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.020 INFO:tasks.ceph:pgid 20.16 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.020 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T12:17:59.043368+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=17, tm_sec=59, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.021 INFO:tasks.ceph:pgid 20.15 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.021 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-30T12:17:59.043368+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=17, tm_sec=59, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.021 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-30T12:17:53.592563+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=17, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.021 INFO:tasks.ceph:pgid 20.1f last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.022 INFO:tasks.ceph:pgid 20.1e last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.022 INFO:tasks.ceph:pgid 20.1d last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.022 INFO:tasks.ceph:pgid 20.1c last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.023 INFO:tasks.ceph:pgid 20.1b last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.023 INFO:tasks.ceph:pgid 20.1a last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.023 INFO:tasks.ceph:pgid 20.19 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.024 INFO:tasks.ceph:pgid 20.18 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.024 INFO:tasks.ceph:pgid 20.7 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.024 INFO:tasks.ceph:pgid 20.6 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.024 INFO:tasks.ceph:pgid 20.5 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.025 INFO:tasks.ceph:pgid 20.4 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:39:55.025 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:39:57.424 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:12.497 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:57.427 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:12.500 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:57.442 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:12.515 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:57.471 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:12.518 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:57.473 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:12.520 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:57.474 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:12.548 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:57.476 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:12.550 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:57.483 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:12.530 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:57.489 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:12.563 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:07.456 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:22.530 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:07.463 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:22.536 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:07.482 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:22.555 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:07.505 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:22.553 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:07.508 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:22.556 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:07.514 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:22.587 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:07.516 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:22.590 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:07.522 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:22.570 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:07.529 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:22.603 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:15.026 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:40:15.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.133+0000 7f978b89b640 10 monclient: get_monmap_and_config 2023-10-30T12:40:15.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.133+0000 7f978b89b640 10 monclient: build_initial_monmap 2023-10-30T12:40:15.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.133+0000 7f978b89b640 10 monclient: monmap: 2023-10-30T12:40:15.136 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:40:15.136 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:15.136 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:40:15.134976+0000 2023-10-30T12:40:15.137 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:40:15.134976+0000 2023-10-30T12:40:15.137 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:40:15.137 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:15.137 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:40:15.137 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:40:15.137 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:40:15.138 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:40:15.138 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:40:15.138 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:15.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.134+0000 7f978b89b640 1 Processor -- start 2023-10-30T12:40:15.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.134+0000 7f978b89b640 1 -- start start 2023-10-30T12:40:15.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.134+0000 7f978b89b640 10 monclient: init 2023-10-30T12:40:15.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 10 monclient: _reopen_session rank -1 2023-10-30T12:40:15.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-10-30T12:40:15.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97840aeb40 0x7f97840ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 10 monclient(hunting): picked mon.noname-c con 0x7f97840aeb40 addr v2:172.21.15.150:3300/0 2023-10-30T12:40:15.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97840aabc0 addr v1:172.21.15.150:6789/0 2023-10-30T12:40:15.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f97840ab040 0x7f97840acde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 10 monclient(hunting): picked mon.noname-a con 0x7f97840ab040 addr v2:172.21.15.148:3300/0 2023-10-30T12:40:15.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97840ac130 con 0x7f97840aabc0 2023-10-30T12:40:15.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:15.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9784067340 con 0x7f97840ab040 2023-10-30T12:40:15.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:15.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97840aef20 con 0x7f97840aeb40 2023-10-30T12:40:15.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 10 monclient(hunting): _renew_subs 2023-10-30T12:40:15.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f978b89b640 10 monclient(hunting): authenticate will time out at 2907.340163s 2023-10-30T12:40:15.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f9789e38640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f97840ab040 0x7f97840acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.135+0000 7f9789637640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97840aeb40 0x7f97840ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789e38640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f97840ab040 0x7f97840acde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55806/0 (socket says 172.21.15.148:55806) 2023-10-30T12:40:15.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789e38640 1 -- 172.21.15.148:0/3036592229 learned_addr learned my addr 172.21.15.148:0/3036592229 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:40:15.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789e38640 10 monclient(hunting): get_auth_request con 0x7f97840ab040 auth_method 0 2023-10-30T12:40:15.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789e38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:15.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789e38640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:15.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789e38640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:15.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789637640 10 monclient(hunting): get_auth_request con 0x7f97840aeb40 auth_method 0 2023-10-30T12:40:15.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789637640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:15.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789637640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:15.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789637640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:15.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789e38640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:15.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789e38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:15.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789e38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:15.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789637640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:15.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789637640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:15.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.136+0000 7f9789637640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:15.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.138+0000 7f9789e38640 10 monclient(hunting): handle_auth_done global_id 16323 payload 274 2023-10-30T12:40:15.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.138+0000 7f9789e38640 1 -- 172.21.15.148:0/3036592229 >> v2:172.21.15.150:3300/0 conn(0x7f97840aeb40 msgr2=0x7f97840ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:15.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.138+0000 7f9789e38640 1 --2- 172.21.15.148:0/3036592229 >> v2:172.21.15.150:3300/0 conn(0x7f97840aeb40 0x7f97840ac8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.138+0000 7f9789e38640 1 -- 172.21.15.148:0/3036592229 >> v1:172.21.15.150:6789/0 conn(0x7f97840aabc0 legacy=0x7f978413b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:15.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.138+0000 7f9789e38640 10 monclient: _finish_hunting 0 2023-10-30T12:40:15.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.138+0000 7f9789e38640 1 monclient: found mon.noname-a 2023-10-30T12:40:15.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.138+0000 7f9789e38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:40:15.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.138+0000 7f9789e38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:40:15.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.138+0000 7f9789e38640 1 -- 172.21.15.148:0/3036592229 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97840aa6a0 con 0x7f97840ab040 2023-10-30T12:40:15.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.139+0000 7f9789e38640 1 --2- 172.21.15.148:0/3036592229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840ab040 0x7f97840acde0 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7f978406bfc0 tx=0x7f977c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306e892c90262157 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:15.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 1 -- 172.21.15.148:0/3036592229 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c02c960 con 0x7f97840ab040 2023-10-30T12:40:15.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:15.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:40:15.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 10 monclient: dump: 2023-10-30T12:40:15.153 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:15.153 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:15.153 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:15.153 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:15.154 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:15.154 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:15.154 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:15.154 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:15.154 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:15.155 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:15.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 10 monclient: _finish_auth 0 2023-10-30T12:40:15.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:40:15.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:40:15.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 1 -- 172.21.15.148:0/3036592229 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c02cad0 con 0x7f97840ab040 2023-10-30T12:40:15.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f978b89b640 5 monclient: authenticate success, global_id 16323 2023-10-30T12:40:15.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f978b89b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:40:15.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:40:15.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f978b89b640 10 monclient: get_monmap_and_config success 2023-10-30T12:40:15.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f978b89b640 4 set_mon_vals no callback set 2023-10-30T12:40:15.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 1 -- 172.21.15.148:0/3036592229 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c02ce70 con 0x7f97840ab040 2023-10-30T12:40:15.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:15.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:40:15.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.143+0000 7f977bfff640 10 monclient: dump: 2023-10-30T12:40:15.157 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:15.157 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:15.157 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:15.157 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:15.158 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:15.158 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:15.158 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:15.158 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:15.158 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:15.159 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:15.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 10 monclient: shutdown 2023-10-30T12:40:15.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:40:15.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 1 -- 172.21.15.148:0/3036592229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840ab040 msgr2=0x7f97840acde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 1 --2- 172.21.15.148:0/3036592229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840ab040 0x7f97840acde0 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7f978406bfc0 tx=0x7f977c02bdc0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 1 -- 172.21.15.148:0/3036592229 shutdown_connections 2023-10-30T12:40:15.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 1 --2- 172.21.15.148:0/3036592229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840ab040 0x7f97840acde0 unknown :-1 s=CLOSED pgs=3241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 1 --2- 172.21.15.148:0/3036592229 >> v2:172.21.15.150:3300/0 conn(0x7f97840aeb40 0x7f97840ac8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 1 -- 172.21.15.148:0/3036592229 >> 172.21.15.148:0/3036592229 conn(0x7f97840b3260 msgr2=0x7f97840b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:15.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 1 -- 172.21.15.148:0/3036592229 shutdown_connections 2023-10-30T12:40:15.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 1 -- 172.21.15.148:0/3036592229 wait complete. 2023-10-30T12:40:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 10 monclient: build_initial_monmap 2023-10-30T12:40:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.144+0000 7f978b89b640 10 monclient: monmap: 2023-10-30T12:40:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:40:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:40:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:40:15.161 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:40:15.162 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:40:15.162 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:40:15.162 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:40:15.162 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:40:15.162 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:40:15.163 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 librados: starting msgr at 2023-10-30T12:40:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 librados: starting objecter 2023-10-30T12:40:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 Processor -- start 2023-10-30T12:40:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 -- start start 2023-10-30T12:40:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 librados: setting wanted keys 2023-10-30T12:40:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 librados: calling monclient init 2023-10-30T12:40:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient: init 2023-10-30T12:40:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient: _reopen_session rank -1 2023-10-30T12:40:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:40:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f97840aabc0 0x7f97841d7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient(hunting): picked mon.noname-c con 0x7f97840aabc0 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:40:15.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97840ab040 0x7f978406fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient(hunting): picked mon.noname-b con 0x7f97840ab040 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:40:15.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840aeb40 0x7f978406ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient(hunting): picked mon.noname-a con 0x7f97840aeb40 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:40:15.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:15.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978413ba60 con 0x7f97840aeb40 2023-10-30T12:40:15.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:15.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978413d330 con 0x7f97840ab040 2023-10-30T12:40:15.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:15.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978413c6a0 con 0x7f97840aabc0 2023-10-30T12:40:15.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient(hunting): _renew_subs 2023-10-30T12:40:15.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.145+0000 7f978b89b640 10 monclient(hunting): authenticate will time out at 2907.350800s 2023-10-30T12:40:15.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9789e38640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840aeb40 0x7f978406ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9789e38640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840aeb40 0x7f978406ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55818/0 (socket says 172.21.15.148:55818) 2023-10-30T12:40:15.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9789637640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f97840aabc0 0x7f97841d7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9789e38640 1 -- 172.21.15.148:0/1076351572 learned_addr learned my addr 172.21.15.148:0/1076351572 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:40:15.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9789e38640 10 monclient(hunting): get_auth_request con 0x7f97840aeb40 auth_method 0 2023-10-30T12:40:15.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9789e38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:15.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9789e38640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:15.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9789e38640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:15.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9788e36640 1 --2- 172.21.15.148:0/1076351572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97840ab040 0x7f978406fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9788e36640 10 monclient(hunting): get_auth_request con 0x7f97840ab040 auth_method 0 2023-10-30T12:40:15.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9788e36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:15.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9788e36640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:15.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.146+0000 7f9788e36640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:15.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9789637640 10 monclient(hunting): get_auth_request con 0x7f97840aabc0 auth_method 0 2023-10-30T12:40:15.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9789637640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:15.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9789637640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:15.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9789637640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:15.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9789e38640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:15.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9789e38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:15.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9789e38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:15.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9788e36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:15.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9788e36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9788e36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9789637640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9789637640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.147+0000 7f9789637640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.149+0000 7f9789e38640 10 monclient(hunting): handle_auth_done global_id 16326 payload 824 2023-10-30T12:40:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.149+0000 7f9789e38640 1 -- 172.21.15.148:0/1076351572 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f97840aabc0 msgr2=0x7f97841d7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.149+0000 7f9789e38640 1 --2- 172.21.15.148:0/1076351572 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f97840aabc0 0x7f97841d7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.149+0000 7f9789e38640 1 -- 172.21.15.148:0/1076351572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97840ab040 msgr2=0x7f978406fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.149+0000 7f9789e38640 1 --2- 172.21.15.148:0/1076351572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97840ab040 0x7f978406fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.149+0000 7f9789e38640 10 monclient: _finish_hunting 0 2023-10-30T12:40:15.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.149+0000 7f9789e38640 1 monclient: found mon.noname-a 2023-10-30T12:40:15.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.149+0000 7f9789e38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:40:15.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.149+0000 7f9789e38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:40:15.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.149+0000 7f9789e38640 1 -- 172.21.15.148:0/1076351572 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978413e6b0 con 0x7f97840aeb40 2023-10-30T12:40:15.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.152+0000 7f9789e38640 1 --2- 172.21.15.148:0/1076351572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840aeb40 0x7f978406ffc0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7f977c0093e0 tx=0x7f977c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:15.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.152+0000 7f9779ffb640 10 client.?.objecter ms_handle_connect 0x7f97840aeb40 2023-10-30T12:40:15.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.152+0000 7f9779ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:40:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 1 -- 172.21.15.148:0/1076351572 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c007930 con 0x7f97840aeb40 2023-10-30T12:40:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:40:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 10 monclient: dump: 2023-10-30T12:40:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:15.176 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:15.176 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:15.176 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:15.176 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:15.176 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:15.177 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:15.177 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:15.177 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:15.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 10 monclient: _finish_auth 0 2023-10-30T12:40:15.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:40:15.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:40:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 1 -- 172.21.15.148:0/1076351572 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c007ac0 con 0x7f97840aeb40 2023-10-30T12:40:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f978b89b640 5 monclient: authenticate success, global_id 16326 2023-10-30T12:40:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:40:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 1 -- 172.21.15.148:0/1076351572 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c007e80 con 0x7f97840aeb40 2023-10-30T12:40:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:40:15.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f9779ffb640 10 monclient: dump: 2023-10-30T12:40:15.179 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:15.179 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:15.179 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:15.179 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:15.179 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:15.180 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:15.180 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:15.180 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:15.180 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:15.180 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:15.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f978b89b640 10 monclient: _renew_subs 2023-10-30T12:40:15.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f977b7fe640 4 set_mon_vals no callback set 2023-10-30T12:40:15.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f978b89b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:40:15.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f978b89b640 1 -- 172.21.15.148:0/1076351572 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978413e850 con 0x7f97840aeb40 2023-10-30T12:40:15.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f978b89b640 10 client.16326.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:40:15.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f978b89b640 10 monclient: _renew_subs 2023-10-30T12:40:15.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f978b89b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:40:15.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f978b89b640 1 -- 172.21.15.148:0/1076351572 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97841d7db0 con 0x7f97840aeb40 2023-10-30T12:40:15.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.155+0000 7f978b89b640 1 librados: init done 2023-10-30T12:40:15.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.156+0000 7f978b89b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:40:15.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.156+0000 7f978b89b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:40:15.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.156+0000 7f978b89b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:40:15.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.156+0000 7f978b89b640 1 -- 172.21.15.148:0/1076351572 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97840b14a0 con 0x7f97840aeb40 2023-10-30T12:40:15.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.174+0000 7f9779ffb640 1 -- 172.21.15.148:0/1076351572 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f977c03e450 con 0x7f97840aeb40 2023-10-30T12:40:15.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.174+0000 7f9779ffb640 1 --2- 172.21.15.148:0/1076351572 >> v2:172.21.15.148:6800/3205090179 conn(0x7f976806c810 0x7f976806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.175+0000 7f9789637640 1 --2- 172.21.15.148:0/1076351572 >> v2:172.21.15.148:6800/3205090179 conn(0x7f976806c810 0x7f976806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.175+0000 7f9789637640 10 monclient: get_auth_request con 0x7f976806c810 auth_method 0 2023-10-30T12:40:15.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.175+0000 7f9779ffb640 10 client.16326.objecter ms_handle_connect 0x7f976806c810 2023-10-30T12:40:15.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.175+0000 7f9789637640 1 --2- 172.21.15.148:0/1076351572 >> v2:172.21.15.148:6800/3205090179 conn(0x7f976806c810 0x7f976806ecd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9780009aa0 tx=0x7f9780009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:15.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.176+0000 7f9779ffb640 1 -- 172.21.15.148:0/1076351572 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(900..900 src has 739..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f977c07a9e0 con 0x7f97840aeb40 2023-10-30T12:40:15.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.176+0000 7f9779ffb640 10 client.16326.objecter ms_dispatch 0x7f9784000b90 osd_map(900..900 src has 739..900) v4 2023-10-30T12:40:15.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.176+0000 7f9779ffb640 3 client.16326.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:40:15.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.176+0000 7f9779ffb640 3 client.16326.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:40:15.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.176+0000 7f9779ffb640 20 client.16326.objecter dump_active .. 0 homeless 2023-10-30T12:40:15.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.411+0000 7f9779ffb640 1 -- 172.21.15.148:0/1076351572 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f977c07ea80 con 0x7f97840aeb40 2023-10-30T12:40:15.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.411+0000 7f9779ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:40:15.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.411+0000 7f9779ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:40:15.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.578+0000 7f978b89b640 1 -- 172.21.15.148:0/1076351572 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9784071670 con 0x7f976806c810 2023-10-30T12:40:15.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.583+0000 7f9779ffb640 1 -- 172.21.15.148:0/1076351572 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152182 (secure 0 0 0) 0x7f9784071670 con 0x7f976806c810 2023-10-30T12:40:15.586 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:40:15.587 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T12:40:15.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 10 librados: watch_flush enter 2023-10-30T12:40:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 10 librados: watch_flush exit 2023-10-30T12:40:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 20 client.16326.objecter shutdown clearing up homeless session... 2023-10-30T12:40:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 10 client.16326.objecter successfully canceled tick 2023-10-30T12:40:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 1 -- 172.21.15.148:0/1076351572 >> v2:172.21.15.148:6800/3205090179 conn(0x7f976806c810 msgr2=0x7f976806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 1 --2- 172.21.15.148:0/1076351572 >> v2:172.21.15.148:6800/3205090179 conn(0x7f976806c810 0x7f976806ecd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9780009aa0 tx=0x7f9780009190 comp rx=0 tx=0).stop 2023-10-30T12:40:15.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 10 monclient: shutdown 2023-10-30T12:40:15.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:40:15.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 1 -- 172.21.15.148:0/1076351572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840aeb40 msgr2=0x7f978406ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 1 --2- 172.21.15.148:0/1076351572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840aeb40 0x7f978406ffc0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7f977c0093e0 tx=0x7f977c004a40 comp rx=0 tx=0).stop 2023-10-30T12:40:15.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 1 -- 172.21.15.148:0/1076351572 shutdown_connections 2023-10-30T12:40:15.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 1 --2- 172.21.15.148:0/1076351572 >> v2:172.21.15.148:6800/3205090179 conn(0x7f976806c810 0x7f976806ecd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 1 --2- 172.21.15.148:0/1076351572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97840aeb40 0x7f978406ffc0 unknown :-1 s=CLOSED pgs=3242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 1 --2- 172.21.15.148:0/1076351572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97840ab040 0x7f978406fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 1 --2- 172.21.15.148:0/1076351572 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f97840aabc0 0x7f97841d7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.585+0000 7f978b89b640 1 -- 172.21.15.148:0/1076351572 >> 172.21.15.148:0/1076351572 conn(0x7f97840b3260 msgr2=0x7f978413ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:15.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.586+0000 7f978b89b640 1 -- 172.21.15.148:0/1076351572 shutdown_connections 2023-10-30T12:40:15.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.586+0000 7f978b89b640 1 -- 172.21.15.148:0/1076351572 wait complete. 2023-10-30T12:40:15.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:15.586+0000 7f978b89b640 1 librados: shutdown 2023-10-30T12:40:15.599 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1577,"stamp":"2023-10-30T12:40:15.316798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":541976,"kb_used_data":70488,"kb_used_omap":12,"kb_used_meta":471411,"kb_avail":754432744,"statfs":{"total":773094113280,"available":772539129856,"internally_reserved":0,"allocated":72179712,"data_stored":36200504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":482725433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002903"},"pg_stats":[{"pgid":"20.8","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:59.863155+0000","last_change":"2023-10-30T12:39:59.864177+0000","last_active":"2023-10-30T12:39:59.863155+0000","last_peered":"2023-10-30T12:39:59.863155+0000","last_clean":"2023-10-30T12:39:59.863155+0000","last_became_active":"2023-10-30T12:19:35.850666+0000","last_became_peered":"2023-10-30T12:19:35.850666+0000","last_unstale":"2023-10-30T12:39:59.863155+0000","last_undegraded":"2023-10-30T12:39:59.863155+0000","last_fullsized":"2023-10-30T12:39:59.863155+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:59.861807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:59.861807+0000","last_clean_scrub_stamp":"2023-10-30T12:39:59.861807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:02.439914+0000","scrub_duration":0.027726994000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:53.854380+0000","last_change":"2023-10-30T12:39:53.859778+0000","last_active":"2023-10-30T12:39:53.854380+0000","last_peered":"2023-10-30T12:39:53.854380+0000","last_clean":"2023-10-30T12:39:53.854380+0000","last_became_active":"2023-10-30T12:19:38.546595+0000","last_became_peered":"2023-10-30T12:19:38.546595+0000","last_unstale":"2023-10-30T12:39:53.854380+0000","last_undegraded":"2023-10-30T12:39:53.854380+0000","last_fullsized":"2023-10-30T12:39:53.854380+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:53.852466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:53.852466+0000","last_clean_scrub_stamp":"2023-10-30T12:39:53.852466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:59.053056+0000","scrub_duration":0.045416113000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:02.886563+0000","last_change":"2023-10-30T12:40:02.887616+0000","last_active":"2023-10-30T12:40:02.886563+0000","last_peered":"2023-10-30T12:40:02.886563+0000","last_clean":"2023-10-30T12:40:02.886563+0000","last_became_active":"2023-10-30T12:19:35.871000+0000","last_became_peered":"2023-10-30T12:19:35.871000+0000","last_unstale":"2023-10-30T12:40:02.886563+0000","last_undegraded":"2023-10-30T12:40:02.886563+0000","last_fullsized":"2023-10-30T12:40:02.886563+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:02.885224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:02.885224+0000","last_clean_scrub_stamp":"2023-10-30T12:40:02.885224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:15.171615+0000","scrub_duration":0.032834037000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":854,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:59.584800+0000","last_change":"2023-10-30T12:39:59.585771+0000","last_active":"2023-10-30T12:39:59.584800+0000","last_peered":"2023-10-30T12:39:59.584800+0000","last_clean":"2023-10-30T12:39:59.584800+0000","last_became_active":"2023-10-30T12:19:29.594505+0000","last_became_peered":"2023-10-30T12:19:29.594505+0000","last_unstale":"2023-10-30T12:39:59.584800+0000","last_undegraded":"2023-10-30T12:39:59.584800+0000","last_fullsized":"2023-10-30T12:39:59.584800+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:59.583409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:59.583409+0000","last_clean_scrub_stamp":"2023-10-30T12:39:59.583409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:47.141085+0000","scrub_duration":0.076237307000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":850,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:47.752324+0000","last_change":"2023-10-30T12:19:27.523347+0000","last_active":"2023-10-30T12:39:47.752324+0000","last_peered":"2023-10-30T12:39:47.752324+0000","last_clean":"2023-10-30T12:39:47.752324+0000","last_became_active":"2023-10-30T12:19:27.515739+0000","last_became_peered":"2023-10-30T12:19:27.515739+0000","last_unstale":"2023-10-30T12:39:47.752324+0000","last_undegraded":"2023-10-30T12:39:47.752324+0000","last_fullsized":"2023-10-30T12:39:47.752324+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":847,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:07.070717+0000","last_change":"2023-10-30T12:40:07.071822+0000","last_active":"2023-10-30T12:40:07.070717+0000","last_peered":"2023-10-30T12:40:07.070717+0000","last_clean":"2023-10-30T12:40:07.070717+0000","last_became_active":"2023-10-30T12:19:31.894071+0000","last_became_peered":"2023-10-30T12:19:31.894071+0000","last_unstale":"2023-10-30T12:40:07.070717+0000","last_undegraded":"2023-10-30T12:40:07.070717+0000","last_fullsized":"2023-10-30T12:40:07.070717+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:07.069366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:07.069366+0000","last_clean_scrub_stamp":"2023-10-30T12:40:07.069366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:49.555106+0000","scrub_duration":0.059461060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:03.248922+0000","last_change":"2023-10-30T12:40:03.249927+0000","last_active":"2023-10-30T12:40:03.248922+0000","last_peered":"2023-10-30T12:40:03.248922+0000","last_clean":"2023-10-30T12:40:03.248922+0000","last_became_active":"2023-10-30T12:19:28.142591+0000","last_became_peered":"2023-10-30T12:19:28.142591+0000","last_unstale":"2023-10-30T12:40:03.248922+0000","last_undegraded":"2023-10-30T12:40:03.248922+0000","last_fullsized":"2023-10-30T12:40:03.248922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:03.247585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:03.247585+0000","last_clean_scrub_stamp":"2023-10-30T12:40:03.247585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:24.330050+0000","scrub_duration":0.032344831999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:58.933952+0000","last_change":"2023-10-30T12:39:58.934987+0000","last_active":"2023-10-30T12:39:58.933952+0000","last_peered":"2023-10-30T12:39:58.933952+0000","last_clean":"2023-10-30T12:39:58.933952+0000","last_became_active":"2023-10-30T12:19:38.643834+0000","last_became_peered":"2023-10-30T12:19:38.643834+0000","last_unstale":"2023-10-30T12:39:58.933952+0000","last_undegraded":"2023-10-30T12:39:58.933952+0000","last_fullsized":"2023-10-30T12:39:58.933952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:58.932666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:58.932666+0000","last_clean_scrub_stamp":"2023-10-30T12:39:58.932666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:07.378282+0000","scrub_duration":0.063031205000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:57.536855+0000","last_change":"2023-10-30T12:39:57.537905+0000","last_active":"2023-10-30T12:39:57.536855+0000","last_peered":"2023-10-30T12:39:57.536855+0000","last_clean":"2023-10-30T12:39:57.536855+0000","last_became_active":"2023-10-30T12:19:29.492043+0000","last_became_peered":"2023-10-30T12:19:29.492043+0000","last_unstale":"2023-10-30T12:39:57.536855+0000","last_undegraded":"2023-10-30T12:39:57.536855+0000","last_fullsized":"2023-10-30T12:39:57.536855+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:57.535422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:57.535422+0000","last_clean_scrub_stamp":"2023-10-30T12:39:57.535422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:15:16.187111+0000","scrub_duration":0.050567743999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1","version":"900'8","reported_seq":854,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:56.071751+0000","last_change":"2023-10-30T12:39:56.071751+0000","last_active":"2023-10-30T12:39:56.071751+0000","last_peered":"2023-10-30T12:39:56.071751+0000","last_clean":"2023-10-30T12:39:56.071751+0000","last_became_active":"2023-10-30T12:19:25.212397+0000","last_became_peered":"2023-10-30T12:19:25.212397+0000","last_unstale":"2023-10-30T12:39:56.071751+0000","last_undegraded":"2023-10-30T12:39:56.071751+0000","last_fullsized":"2023-10-30T12:39:56.071751+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'8","last_scrub_stamp":"2023-10-30T12:39:56.069074+0000","last_deep_scrub":"900'8","last_deep_scrub_stamp":"2023-10-30T12:39:56.069074+0000","last_clean_scrub_stamp":"2023-10-30T12:39:56.069074+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:01.898778+0000","scrub_duration":0.079575325000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":858,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:10.738853+0000","last_change":"2023-10-30T12:40:10.739967+0000","last_active":"2023-10-30T12:40:10.738853+0000","last_peered":"2023-10-30T12:40:10.738853+0000","last_clean":"2023-10-30T12:40:10.738853+0000","last_became_active":"2023-10-30T12:19:29.491706+0000","last_became_peered":"2023-10-30T12:19:29.491706+0000","last_unstale":"2023-10-30T12:40:10.738853+0000","last_undegraded":"2023-10-30T12:40:10.738853+0000","last_fullsized":"2023-10-30T12:40:10.738853+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:10.737405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:10.737405+0000","last_clean_scrub_stamp":"2023-10-30T12:40:10.737405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:27.660864+0000","scrub_duration":0.029906892000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":859,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:02.129087+0000","last_change":"2023-10-30T12:40:02.130087+0000","last_active":"2023-10-30T12:40:02.129087+0000","last_peered":"2023-10-30T12:40:02.129087+0000","last_clean":"2023-10-30T12:40:02.129087+0000","last_became_active":"2023-10-30T12:19:32.988808+0000","last_became_peered":"2023-10-30T12:19:32.988808+0000","last_unstale":"2023-10-30T12:40:02.129087+0000","last_undegraded":"2023-10-30T12:40:02.129087+0000","last_fullsized":"2023-10-30T12:40:02.129087+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:02.127694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:02.127694+0000","last_clean_scrub_stamp":"2023-10-30T12:40:02.127694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:56.157165+0000","scrub_duration":0.067790453000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":858,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:07.137955+0000","last_change":"2023-10-30T12:40:07.139048+0000","last_active":"2023-10-30T12:40:07.137955+0000","last_peered":"2023-10-30T12:40:07.137955+0000","last_clean":"2023-10-30T12:40:07.137955+0000","last_became_active":"2023-10-30T12:19:28.916380+0000","last_became_peered":"2023-10-30T12:19:28.916380+0000","last_unstale":"2023-10-30T12:40:07.137955+0000","last_undegraded":"2023-10-30T12:40:07.137955+0000","last_fullsized":"2023-10-30T12:40:07.137955+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:07.136266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:07.136266+0000","last_clean_scrub_stamp":"2023-10-30T12:40:07.136266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:22.811859+0000","scrub_duration":0.063096992000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":864,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:59.550234+0000","last_change":"2023-10-30T12:39:59.551230+0000","last_active":"2023-10-30T12:39:59.550234+0000","last_peered":"2023-10-30T12:39:59.550234+0000","last_clean":"2023-10-30T12:39:59.550234+0000","last_became_active":"2023-10-30T12:18:01.568787+0000","last_became_peered":"2023-10-30T12:18:01.568787+0000","last_unstale":"2023-10-30T12:39:59.550234+0000","last_undegraded":"2023-10-30T12:39:59.550234+0000","last_fullsized":"2023-10-30T12:39:59.550234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:59.548935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:59.548935+0000","last_clean_scrub_stamp":"2023-10-30T12:39:59.548935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:50:31.515390+0000","scrub_duration":0.034566703999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:53.219198+0000","last_change":"2023-10-30T12:39:53.220177+0000","last_active":"2023-10-30T12:39:53.219198+0000","last_peered":"2023-10-30T12:39:53.219198+0000","last_clean":"2023-10-30T12:39:53.219198+0000","last_became_active":"2023-10-30T12:19:40.940463+0000","last_became_peered":"2023-10-30T12:19:40.940463+0000","last_unstale":"2023-10-30T12:39:53.219198+0000","last_undegraded":"2023-10-30T12:39:53.219198+0000","last_fullsized":"2023-10-30T12:39:53.219198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:53.217887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:53.217887+0000","last_clean_scrub_stamp":"2023-10-30T12:39:53.217887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:20:39.599062+0000","scrub_duration":0.028260459000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":852,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:52.195089+0000","last_change":"2023-10-30T12:39:52.196146+0000","last_active":"2023-10-30T12:39:52.195089+0000","last_peered":"2023-10-30T12:39:52.195089+0000","last_clean":"2023-10-30T12:39:52.195089+0000","last_became_active":"2023-10-30T12:18:01.602190+0000","last_became_peered":"2023-10-30T12:18:01.602190+0000","last_unstale":"2023-10-30T12:39:52.195089+0000","last_undegraded":"2023-10-30T12:39:52.195089+0000","last_fullsized":"2023-10-30T12:39:52.195089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:52.193720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:52.193720+0000","last_clean_scrub_stamp":"2023-10-30T12:39:52.193720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:33:40.108140+0000","scrub_duration":0.043609539000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:47.306561+0000","last_change":"2023-10-30T12:19:40.722588+0000","last_active":"2023-10-30T12:39:47.306561+0000","last_peered":"2023-10-30T12:39:47.306561+0000","last_clean":"2023-10-30T12:39:47.306561+0000","last_became_active":"2023-10-30T12:19:40.713186+0000","last_became_peered":"2023-10-30T12:19:40.713186+0000","last_unstale":"2023-10-30T12:39:47.306561+0000","last_undegraded":"2023-10-30T12:39:47.306561+0000","last_fullsized":"2023-10-30T12:39:47.306561+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":863,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:06.663966+0000","last_change":"2023-10-30T12:40:06.665005+0000","last_active":"2023-10-30T12:40:06.663966+0000","last_peered":"2023-10-30T12:40:06.663966+0000","last_clean":"2023-10-30T12:40:06.663966+0000","last_became_active":"2023-10-30T12:18:01.051697+0000","last_became_peered":"2023-10-30T12:18:01.051697+0000","last_unstale":"2023-10-30T12:40:06.663966+0000","last_undegraded":"2023-10-30T12:40:06.663966+0000","last_fullsized":"2023-10-30T12:40:06.663966+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:06.662527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:06.662527+0000","last_clean_scrub_stamp":"2023-10-30T12:40:06.662527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:00:24.353609+0000","scrub_duration":0.032173303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":838,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:57.002601+0000","last_change":"2023-10-30T12:39:57.003604+0000","last_active":"2023-10-30T12:39:57.002601+0000","last_peered":"2023-10-30T12:39:57.002601+0000","last_clean":"2023-10-30T12:39:57.002601+0000","last_became_active":"2023-10-30T12:19:40.864175+0000","last_became_peered":"2023-10-30T12:19:40.864175+0000","last_unstale":"2023-10-30T12:39:57.002601+0000","last_undegraded":"2023-10-30T12:39:57.002601+0000","last_fullsized":"2023-10-30T12:39:57.002601+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:57.001273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:57.001273+0000","last_clean_scrub_stamp":"2023-10-30T12:39:57.001273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:41:07.773438+0000","scrub_duration":0.044252685999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":853,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:56.997999+0000","last_change":"2023-10-30T12:39:56.999005+0000","last_active":"2023-10-30T12:39:56.997999+0000","last_peered":"2023-10-30T12:39:56.997999+0000","last_clean":"2023-10-30T12:39:56.997999+0000","last_became_active":"2023-10-30T12:18:01.523269+0000","last_became_peered":"2023-10-30T12:18:01.523269+0000","last_unstale":"2023-10-30T12:39:56.997999+0000","last_undegraded":"2023-10-30T12:39:56.997999+0000","last_fullsized":"2023-10-30T12:39:56.997999+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:56.996587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:56.996587+0000","last_clean_scrub_stamp":"2023-10-30T12:39:56.996587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:44:13.299852+0000","scrub_duration":0.031392213000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:03.086676+0000","last_change":"2023-10-30T12:40:03.088506+0000","last_active":"2023-10-30T12:40:03.086676+0000","last_peered":"2023-10-30T12:40:03.086676+0000","last_clean":"2023-10-30T12:40:03.086676+0000","last_became_active":"2023-10-30T12:19:40.440094+0000","last_became_peered":"2023-10-30T12:19:40.440094+0000","last_unstale":"2023-10-30T12:40:03.086676+0000","last_undegraded":"2023-10-30T12:40:03.086676+0000","last_fullsized":"2023-10-30T12:40:03.086676+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:03.085346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:03.085346+0000","last_clean_scrub_stamp":"2023-10-30T12:40:03.085346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:49:12.994355+0000","scrub_duration":0.038565555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":864,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:55.503565+0000","last_change":"2023-10-30T12:39:55.504537+0000","last_active":"2023-10-30T12:39:55.503565+0000","last_peered":"2023-10-30T12:39:55.503565+0000","last_clean":"2023-10-30T12:39:55.503565+0000","last_became_active":"2023-10-30T12:18:01.504543+0000","last_became_peered":"2023-10-30T12:18:01.504543+0000","last_unstale":"2023-10-30T12:39:55.503565+0000","last_undegraded":"2023-10-30T12:39:55.503565+0000","last_fullsized":"2023-10-30T12:39:55.503565+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:55.502243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:55.502243+0000","last_clean_scrub_stamp":"2023-10-30T12:39:55.502243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:59.951192+0000","scrub_duration":0.037490758999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:54.468480+0000","last_change":"2023-10-30T12:39:54.469537+0000","last_active":"2023-10-30T12:39:54.468480+0000","last_peered":"2023-10-30T12:39:54.468480+0000","last_clean":"2023-10-30T12:39:54.468480+0000","last_became_active":"2023-10-30T12:19:29.543275+0000","last_became_peered":"2023-10-30T12:19:29.543275+0000","last_unstale":"2023-10-30T12:39:54.468480+0000","last_undegraded":"2023-10-30T12:39:54.468480+0000","last_fullsized":"2023-10-30T12:39:54.468480+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:54.467054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:54.467054+0000","last_clean_scrub_stamp":"2023-10-30T12:39:54.467054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:07:39.095292+0000","scrub_duration":0.057546617000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":865,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:00.236189+0000","last_change":"2023-10-30T12:40:00.246472+0000","last_active":"2023-10-30T12:40:00.236189+0000","last_peered":"2023-10-30T12:40:00.236189+0000","last_clean":"2023-10-30T12:40:00.236189+0000","last_became_active":"2023-10-30T12:18:00.904525+0000","last_became_peered":"2023-10-30T12:18:00.904525+0000","last_unstale":"2023-10-30T12:40:00.236189+0000","last_undegraded":"2023-10-30T12:40:00.236189+0000","last_fullsized":"2023-10-30T12:40:00.236189+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:00.232185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:00.232185+0000","last_clean_scrub_stamp":"2023-10-30T12:40:00.232185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:35:47.818510+0000","scrub_duration":0.18464655599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":838,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:00.809299+0000","last_change":"2023-10-30T12:40:00.810355+0000","last_active":"2023-10-30T12:40:00.809299+0000","last_peered":"2023-10-30T12:40:00.809299+0000","last_clean":"2023-10-30T12:40:00.809299+0000","last_became_active":"2023-10-30T12:19:26.061689+0000","last_became_peered":"2023-10-30T12:19:26.061689+0000","last_unstale":"2023-10-30T12:40:00.809299+0000","last_undegraded":"2023-10-30T12:40:00.809299+0000","last_fullsized":"2023-10-30T12:40:00.809299+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:00.807878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:00.807878+0000","last_clean_scrub_stamp":"2023-10-30T12:40:00.807878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:54.022399+0000","scrub_duration":0.043296240999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":865,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:53.499813+0000","last_change":"2023-10-30T12:39:53.499813+0000","last_active":"2023-10-30T12:39:53.499813+0000","last_peered":"2023-10-30T12:39:53.499813+0000","last_clean":"2023-10-30T12:39:53.499813+0000","last_became_active":"2023-10-30T12:18:01.003382+0000","last_became_peered":"2023-10-30T12:18:01.003382+0000","last_unstale":"2023-10-30T12:39:53.499813+0000","last_undegraded":"2023-10-30T12:39:53.499813+0000","last_fullsized":"2023-10-30T12:39:53.499813+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:53.493713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:53.493713+0000","last_clean_scrub_stamp":"2023-10-30T12:39:53.493713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:45.675107+0000","scrub_duration":0.031685023999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":838,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:53.995696+0000","last_change":"2023-10-30T12:39:53.996734+0000","last_active":"2023-10-30T12:39:53.995696+0000","last_peered":"2023-10-30T12:39:53.995696+0000","last_clean":"2023-10-30T12:39:53.995696+0000","last_became_active":"2023-10-30T12:19:25.272220+0000","last_became_peered":"2023-10-30T12:19:25.272220+0000","last_unstale":"2023-10-30T12:39:53.995696+0000","last_undegraded":"2023-10-30T12:39:53.995696+0000","last_fullsized":"2023-10-30T12:39:53.995696+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:53.994259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:53.994259+0000","last_clean_scrub_stamp":"2023-10-30T12:39:53.994259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:25.256027+0000","scrub_duration":0.091879046000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":864,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:57.570989+0000","last_change":"2023-10-30T12:39:57.573801+0000","last_active":"2023-10-30T12:39:57.570989+0000","last_peered":"2023-10-30T12:39:57.570989+0000","last_clean":"2023-10-30T12:39:57.570989+0000","last_became_active":"2023-10-30T12:18:01.393697+0000","last_became_peered":"2023-10-30T12:18:01.393697+0000","last_unstale":"2023-10-30T12:39:57.570989+0000","last_undegraded":"2023-10-30T12:39:57.570989+0000","last_fullsized":"2023-10-30T12:39:57.570989+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:57.566568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:57.566568+0000","last_clean_scrub_stamp":"2023-10-30T12:39:57.566568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:41:06.942534+0000","scrub_duration":0.048101698999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"900'78","reported_seq":955,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:52.977088+0000","last_change":"2023-10-30T12:39:52.977088+0000","last_active":"2023-10-30T12:39:52.977088+0000","last_peered":"2023-10-30T12:39:52.977088+0000","last_clean":"2023-10-30T12:39:52.977088+0000","last_became_active":"2023-10-30T12:17:56.575838+0000","last_became_peered":"2023-10-30T12:17:56.575838+0000","last_unstale":"2023-10-30T12:39:52.977088+0000","last_undegraded":"2023-10-30T12:39:52.977088+0000","last_fullsized":"2023-10-30T12:39:52.977088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'78","last_scrub_stamp":"2023-10-30T12:39:52.973839+0000","last_deep_scrub":"900'78","last_deep_scrub_stamp":"2023-10-30T12:39:52.973839+0000","last_clean_scrub_stamp":"2023-10-30T12:39:52.973839+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:33:03.900796+0000","scrub_duration":0.218512285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":851,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:50.894409+0000","last_change":"2023-10-30T12:19:25.736207+0000","last_active":"2023-10-30T12:39:50.894409+0000","last_peered":"2023-10-30T12:39:50.894409+0000","last_clean":"2023-10-30T12:39:50.894409+0000","last_became_active":"2023-10-30T12:19:25.715885+0000","last_became_peered":"2023-10-30T12:19:25.715885+0000","last_unstale":"2023-10-30T12:39:50.894409+0000","last_undegraded":"2023-10-30T12:39:50.894409+0000","last_fullsized":"2023-10-30T12:39:50.894409+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":852,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:02.539109+0000","last_change":"2023-10-30T12:40:02.540166+0000","last_active":"2023-10-30T12:40:02.539109+0000","last_peered":"2023-10-30T12:40:02.539109+0000","last_clean":"2023-10-30T12:40:02.539109+0000","last_became_active":"2023-10-30T12:19:29.879226+0000","last_became_peered":"2023-10-30T12:19:29.879226+0000","last_unstale":"2023-10-30T12:40:02.539109+0000","last_undegraded":"2023-10-30T12:40:02.539109+0000","last_fullsized":"2023-10-30T12:40:02.539109+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:02.537572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:02.537572+0000","last_clean_scrub_stamp":"2023-10-30T12:40:02.537572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:52.257173+0000","scrub_duration":0.046849797999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":848,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:47.319738+0000","last_change":"2023-10-30T12:19:40.799718+0000","last_active":"2023-10-30T12:39:47.319738+0000","last_peered":"2023-10-30T12:39:47.319738+0000","last_clean":"2023-10-30T12:39:47.319738+0000","last_became_active":"2023-10-30T12:19:40.794341+0000","last_became_peered":"2023-10-30T12:19:40.794341+0000","last_unstale":"2023-10-30T12:39:47.319738+0000","last_undegraded":"2023-10-30T12:39:47.319738+0000","last_fullsized":"2023-10-30T12:39:47.319738+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:47.316811+0000","last_change":"2023-10-30T12:19:28.247323+0000","last_active":"2023-10-30T12:39:47.316811+0000","last_peered":"2023-10-30T12:39:47.316811+0000","last_clean":"2023-10-30T12:39:47.316811+0000","last_became_active":"2023-10-30T12:19:28.238513+0000","last_became_peered":"2023-10-30T12:19:28.238513+0000","last_unstale":"2023-10-30T12:39:47.316811+0000","last_undegraded":"2023-10-30T12:39:47.316811+0000","last_fullsized":"2023-10-30T12:39:47.316811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:19:17.767568+0000","last_clean_scrub_stamp":"2023-10-30T12:19:17.767568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:55.219738+0000","last_change":"2023-10-30T12:39:55.220724+0000","last_active":"2023-10-30T12:39:55.219738+0000","last_peered":"2023-10-30T12:39:55.219738+0000","last_clean":"2023-10-30T12:39:55.219738+0000","last_became_active":"2023-10-30T12:19:27.284772+0000","last_became_peered":"2023-10-30T12:19:27.284772+0000","last_unstale":"2023-10-30T12:39:55.219738+0000","last_undegraded":"2023-10-30T12:39:55.219738+0000","last_fullsized":"2023-10-30T12:39:55.219738+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:55.218406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:55.218406+0000","last_clean_scrub_stamp":"2023-10-30T12:39:55.218406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:00.385740+0000","scrub_duration":0.055291405000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":838,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:59.840386+0000","last_change":"2023-10-30T12:39:59.841454+0000","last_active":"2023-10-30T12:39:59.840386+0000","last_peered":"2023-10-30T12:39:59.840386+0000","last_clean":"2023-10-30T12:39:59.840386+0000","last_became_active":"2023-10-30T12:19:33.483122+0000","last_became_peered":"2023-10-30T12:19:33.483122+0000","last_unstale":"2023-10-30T12:39:59.840386+0000","last_undegraded":"2023-10-30T12:39:59.840386+0000","last_fullsized":"2023-10-30T12:39:59.840386+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:59.837322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:59.837322+0000","last_clean_scrub_stamp":"2023-10-30T12:39:59.837322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:39.954540+0000","scrub_duration":0.041544445999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":853,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:56.509862+0000","last_change":"2023-10-30T12:39:56.510854+0000","last_active":"2023-10-30T12:39:56.509862+0000","last_peered":"2023-10-30T12:39:56.509862+0000","last_clean":"2023-10-30T12:39:56.509862+0000","last_became_active":"2023-10-30T12:19:26.663005+0000","last_became_peered":"2023-10-30T12:19:26.663005+0000","last_unstale":"2023-10-30T12:39:56.509862+0000","last_undegraded":"2023-10-30T12:39:56.509862+0000","last_fullsized":"2023-10-30T12:39:56.509862+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:56.508524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:56.508524+0000","last_clean_scrub_stamp":"2023-10-30T12:39:56.508524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:48.208342+0000","scrub_duration":0.053320273000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":857,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:07.676399+0000","last_change":"2023-10-30T12:40:07.677510+0000","last_active":"2023-10-30T12:40:07.676399+0000","last_peered":"2023-10-30T12:40:07.676399+0000","last_clean":"2023-10-30T12:40:07.676399+0000","last_became_active":"2023-10-30T12:19:26.766234+0000","last_became_peered":"2023-10-30T12:19:26.766234+0000","last_unstale":"2023-10-30T12:40:07.676399+0000","last_undegraded":"2023-10-30T12:40:07.676399+0000","last_fullsized":"2023-10-30T12:40:07.676399+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:07.674810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:07.674810+0000","last_clean_scrub_stamp":"2023-10-30T12:40:07.674810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:45.104913+0000","scrub_duration":0.034958095000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:56.886448+0000","last_change":"2023-10-30T12:39:56.890537+0000","last_active":"2023-10-30T12:39:56.886448+0000","last_peered":"2023-10-30T12:39:56.886448+0000","last_clean":"2023-10-30T12:39:56.886448+0000","last_became_active":"2023-10-30T12:19:29.191499+0000","last_became_peered":"2023-10-30T12:19:29.191499+0000","last_unstale":"2023-10-30T12:39:56.886448+0000","last_undegraded":"2023-10-30T12:39:56.886448+0000","last_fullsized":"2023-10-30T12:39:56.886448+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:56.885085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:56.885085+0000","last_clean_scrub_stamp":"2023-10-30T12:39:56.885085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:14.236063+0000","scrub_duration":0.059900850999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":859,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:12.154811+0000","last_change":"2023-10-30T12:40:12.155814+0000","last_active":"2023-10-30T12:40:12.154811+0000","last_peered":"2023-10-30T12:40:12.154811+0000","last_clean":"2023-10-30T12:40:12.154811+0000","last_became_active":"2023-10-30T12:19:30.408141+0000","last_became_peered":"2023-10-30T12:19:30.408141+0000","last_unstale":"2023-10-30T12:40:12.154811+0000","last_undegraded":"2023-10-30T12:40:12.154811+0000","last_fullsized":"2023-10-30T12:40:12.154811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:12.153519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:12.153519+0000","last_clean_scrub_stamp":"2023-10-30T12:40:12.153519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:09.258954+0000","scrub_duration":0.045757989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:06.107242+0000","last_change":"2023-10-30T12:40:06.108328+0000","last_active":"2023-10-30T12:40:06.107242+0000","last_peered":"2023-10-30T12:40:06.107242+0000","last_clean":"2023-10-30T12:40:06.107242+0000","last_became_active":"2023-10-30T12:19:40.528642+0000","last_became_peered":"2023-10-30T12:19:40.528642+0000","last_unstale":"2023-10-30T12:40:06.107242+0000","last_undegraded":"2023-10-30T12:40:06.107242+0000","last_fullsized":"2023-10-30T12:40:06.107242+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:06.105884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:06.105884+0000","last_clean_scrub_stamp":"2023-10-30T12:40:06.105884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:37.288711+0000","scrub_duration":0.055356317000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":847,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:59.038655+0000","last_change":"2023-10-30T12:39:59.039684+0000","last_active":"2023-10-30T12:39:59.038655+0000","last_peered":"2023-10-30T12:39:59.038655+0000","last_clean":"2023-10-30T12:39:59.038655+0000","last_became_active":"2023-10-30T12:19:30.720900+0000","last_became_peered":"2023-10-30T12:19:30.720900+0000","last_unstale":"2023-10-30T12:39:59.038655+0000","last_undegraded":"2023-10-30T12:39:59.038655+0000","last_fullsized":"2023-10-30T12:39:59.038655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:59.037328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:59.037328+0000","last_clean_scrub_stamp":"2023-10-30T12:39:59.037328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:06:15.048285+0000","scrub_duration":0.04127786,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509711,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62144,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309696,"statfs":{"total":96636764160,"available":96573128704,"internally_reserved":0,"allocated":8970240,"data_stored":4460959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":23.231999999999999,"5min":17.655999999999999,"15min":16.648},"min":{"1min":6.4530000000000003,"5min":4.0250000000000004,"15min":4.0250000000000004},"max":{"1min":181.905,"5min":233.52500000000001,"15min":233.52500000000001},"last":10.074},{"interface":"front","average":{"1min":27.143000000000001,"5min":21.59,"15min":22.366},"min":{"1min":9.7650000000000006,"5min":5.6740000000000004,"15min":5.6740000000000004},"max":{"1min":183.072,"5min":214.31899999999999,"15min":365.47399999999999},"last":10.763}]},{"osd":1,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.035,"5min":12.183,"15min":14.696},"min":{"1min":6.0800000000000001,"5min":4.4359999999999999,"15min":4.2389999999999999},"max":{"1min":63.286999999999999,"5min":63.286999999999999,"15min":142.51400000000001},"last":9.7599999999999998},{"interface":"front","average":{"1min":17.978999999999999,"5min":14.823,"15min":17.242000000000001},"min":{"1min":6.056,"5min":5.1689999999999996,"15min":5.1669999999999998},"max":{"1min":61.552,"5min":112.276,"15min":211.11600000000001},"last":14.003}]},{"osd":2,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.715,"5min":21.622,"15min":20.561},"min":{"1min":6.5259999999999998,"5min":6.2530000000000001,"15min":6.2530000000000001},"max":{"1min":61.927,"5min":187.00399999999999,"15min":187.00399999999999},"last":18.486000000000001},{"interface":"front","average":{"1min":20.795999999999999,"5min":23.681999999999999,"15min":21.565000000000001},"min":{"1min":8.2710000000000008,"5min":6.5359999999999996,"15min":5.6260000000000003},"max":{"1min":80.686999999999998,"5min":240.99799999999999,"15min":240.99799999999999},"last":17.716000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":20.48,"5min":26.420000000000002,"15min":25.207999999999998},"min":{"1min":10.49,"5min":7.5990000000000002,"15min":6.2240000000000002},"max":{"1min":62.517000000000003,"5min":279.94099999999997,"15min":279.94099999999997},"last":16.332000000000001},{"interface":"front","average":{"1min":17.039000000000001,"5min":24.077999999999999,"15min":23.585000000000001},"min":{"1min":8.1370000000000005,"5min":6.617,"15min":6.5960000000000001},"max":{"1min":55.351999999999997,"5min":180.74600000000001,"15min":180.74600000000001},"last":15.063000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":13.237,"5min":19.042000000000002,"15min":18.274999999999999},"min":{"1min":6.617,"5min":6.617,"15min":6.1840000000000002},"max":{"1min":59.103000000000002,"5min":184.88,"15min":184.88},"last":11.894},{"interface":"front","average":{"1min":20.768999999999998,"5min":26.350000000000001,"15min":25.687999999999999},"min":{"1min":11.442,"5min":9.4160000000000004,"15min":7.3540000000000001},"max":{"1min":54.426000000000002,"5min":374.38900000000001,"15min":374.38900000000001},"last":13.324999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":19.882000000000001,"5min":25.530999999999999,"15min":25.718},"min":{"1min":10.507,"5min":8.1210000000000004,"15min":6.2729999999999997},"max":{"1min":43.252000000000002,"5min":218.28800000000001,"15min":218.28800000000001},"last":15.672000000000001},{"interface":"front","average":{"1min":17.363,"5min":25.651,"15min":23.794},"min":{"1min":6.7160000000000002,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":34.281999999999996,"5min":195.74700000000001,"15min":195.74700000000001},"last":14.662000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":21.611999999999998,"5min":23.324000000000002,"15min":27.591999999999999},"min":{"1min":9.8620000000000001,"5min":9.593,"15min":8.3320000000000007},"max":{"1min":69.704999999999998,"5min":178.68100000000001,"15min":224.71000000000001},"last":17.093},{"interface":"front","average":{"1min":19.506,"5min":24.533000000000001,"15min":25.074000000000002},"min":{"1min":9.7799999999999994,"5min":7.2469999999999999,"15min":7.2469999999999999},"max":{"1min":39.325000000000003,"5min":187.14500000000001,"15min":210.05199999999999},"last":20.855}]}]},{"osd":5,"up_from":15,"seq":64424509711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80780,"kb_used_data":9348,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94291060,"statfs":{"total":96636764160,"available":96554045440,"internally_reserved":0,"allocated":9572352,"data_stored":4982006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:46 2023","interfaces":[{"interface":"back","average":{"1min":13.422000000000001,"5min":16.344999999999999,"15min":23.686},"min":{"1min":4.9379999999999997,"5min":4.1050000000000004,"15min":4.1050000000000004},"max":{"1min":61.497999999999998,"5min":158.87799999999999,"15min":308.51999999999998},"last":9.4450000000000003},{"interface":"front","average":{"1min":15.584,"5min":17.266999999999999,"15min":22.806999999999999},"min":{"1min":5.7670000000000003,"5min":4.7469999999999999,"15min":4.7469999999999999},"max":{"1min":66.950999999999993,"5min":146.374,"15min":244.072},"last":5.7539999999999996}]},{"osd":1,"last update":"Mon Oct 30 12:39:56 2023","interfaces":[{"interface":"back","average":{"1min":24.869,"5min":18.227,"15min":24.437000000000001},"min":{"1min":4.9160000000000004,"5min":4.9160000000000004,"15min":4.9160000000000004},"max":{"1min":227.51499999999999,"5min":227.51499999999999,"15min":304.13299999999998},"last":8.6329999999999991},{"interface":"front","average":{"1min":22.501999999999999,"5min":18.318000000000001,"15min":23.754999999999999},"min":{"1min":7.8200000000000003,"5min":5.6150000000000002,"15min":5.4109999999999996},"max":{"1min":102.226,"5min":162.25200000000001,"15min":278.33999999999997},"last":12.193}]},{"osd":2,"last update":"Mon Oct 30 12:39:56 2023","interfaces":[{"interface":"back","average":{"1min":26.420000000000002,"5min":20.91,"15min":22.443000000000001},"min":{"1min":5.6989999999999998,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":219.33600000000001,"5min":219.33600000000001,"15min":246.37},"last":10.827999999999999},{"interface":"front","average":{"1min":31.077999999999999,"5min":24.548999999999999,"15min":23.369},"min":{"1min":9.3379999999999992,"5min":8.4540000000000006,"15min":5.2770000000000001},"max":{"1min":229.303,"5min":229.303,"15min":303.274},"last":12.811999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:39:26 2023","interfaces":[{"interface":"back","average":{"1min":19.34,"5min":23.516999999999999,"15min":25.564},"min":{"1min":6.8339999999999996,"5min":6.3860000000000001,"15min":6.3860000000000001},"max":{"1min":97.822999999999993,"5min":148.125,"15min":273.10300000000001},"last":15.172000000000001},{"interface":"front","average":{"1min":24.561,"5min":21.800000000000001,"15min":24.858000000000001},"min":{"1min":9.3970000000000002,"5min":6.2889999999999997,"15min":5.8029999999999999},"max":{"1min":94.296000000000006,"5min":154.857,"15min":282.28500000000003},"last":14.536}]},{"osd":4,"last update":"Mon Oct 30 12:39:56 2023","interfaces":[{"interface":"back","average":{"1min":20.756,"5min":22.402999999999999,"15min":27.849},"min":{"1min":11.57,"5min":9.5109999999999992,"15min":7.2830000000000004},"max":{"1min":92.058000000000007,"5min":144.102,"15min":338.60000000000002},"last":13.375999999999999},{"interface":"front","average":{"1min":17.724,"5min":24.602,"15min":27.478999999999999},"min":{"1min":7.4420000000000002,"5min":7.4420000000000002,"15min":6.3890000000000002},"max":{"1min":97.159999999999997,"5min":169.45500000000001,"15min":295.76600000000002},"last":7.7089999999999996}]},{"osd":6,"last update":"Mon Oct 30 12:39:56 2023","interfaces":[{"interface":"back","average":{"1min":21.257000000000001,"5min":25.925999999999998,"15min":28.923999999999999},"min":{"1min":10.871,"5min":10.871,"15min":7.9729999999999999},"max":{"1min":94.408000000000001,"5min":178.637,"15min":255.291},"last":14.002000000000001},{"interface":"front","average":{"1min":24.161999999999999,"5min":31.087,"15min":34.582999999999998},"min":{"1min":13.163,"5min":12.661,"15min":7.5430000000000001},"max":{"1min":95.777000000000001,"5min":319.17700000000002,"15min":320.678},"last":16.506}]},{"osd":7,"last update":"Mon Oct 30 12:39:26 2023","interfaces":[{"interface":"back","average":{"1min":20.66,"5min":27.561,"15min":28.265999999999998},"min":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"max":{"1min":113.35899999999999,"5min":258.61200000000002,"15min":258.61200000000002},"last":10.098000000000001},{"interface":"front","average":{"1min":16.838999999999999,"5min":22.385999999999999,"15min":24.34},"min":{"1min":6.109,"5min":6.109,"15min":6.109},"max":{"1min":55.759,"5min":145.15000000000001,"15min":245.71899999999999},"last":15.824}]}]},{"osd":4,"up_from":15,"seq":64424509712,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62144,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309696,"statfs":{"total":96636764160,"available":96573128704,"internally_reserved":0,"allocated":8970240,"data_stored":4460959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":8.8330000000000002,"5min":15.872,"15min":22.242999999999999},"min":{"1min":4.7569999999999997,"5min":4.2140000000000004,"15min":4.0620000000000003},"max":{"1min":16.516999999999999,"5min":158.499,"15min":295.56400000000002},"last":7.5090000000000003},{"interface":"front","average":{"1min":8.3000000000000007,"5min":18.66,"15min":23.565999999999999},"min":{"1min":4.4080000000000004,"5min":4.4080000000000004,"15min":4.4080000000000004},"max":{"1min":14.081,"5min":182.96600000000001,"15min":368.01900000000001},"last":7.0780000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:39:46 2023","interfaces":[{"interface":"back","average":{"1min":11.259,"5min":16.731000000000002,"15min":20.905999999999999},"min":{"1min":6.633,"5min":5.6420000000000003,"15min":4.5389999999999997},"max":{"1min":18.992999999999999,"5min":171.24100000000001,"15min":322.97800000000001},"last":7.8739999999999997},{"interface":"front","average":{"1min":15.079000000000001,"5min":21.649000000000001,"15min":20.692},"min":{"1min":6.6539999999999999,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":32.031999999999996,"5min":168.833,"15min":245.44900000000001},"last":14.989000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:39:54 2023","interfaces":[{"interface":"back","average":{"1min":15.304,"5min":26.896999999999998,"15min":23.456},"min":{"1min":6.8570000000000002,"5min":6.641,"15min":6.0039999999999996},"max":{"1min":60.234999999999999,"5min":316.74799999999999,"15min":316.74799999999999},"last":8.4800000000000004},{"interface":"front","average":{"1min":17.652999999999999,"5min":26.408000000000001,"15min":25.983000000000001},"min":{"1min":6.7469999999999999,"5min":5.9489999999999998,"15min":5.9489999999999998},"max":{"1min":52.188000000000002,"5min":317.55399999999997,"15min":317.55399999999997},"last":9.1989999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":14.321999999999999,"5min":23.728999999999999,"15min":28.404},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":6.7270000000000003},"max":{"1min":21.91,"5min":161.197,"15min":383.87099999999998},"last":14.397},{"interface":"front","average":{"1min":13.757999999999999,"5min":26.026,"15min":26.616},"min":{"1min":8.5,"5min":8.5,"15min":6.6529999999999996},"max":{"1min":18.109999999999999,"5min":260.94200000000001,"15min":284.61700000000002},"last":11.868}]},{"osd":5,"last update":"Mon Oct 30 12:39:46 2023","interfaces":[{"interface":"back","average":{"1min":19.042999999999999,"5min":27.260999999999999,"15min":27.414000000000001},"min":{"1min":8.2910000000000004,"5min":7.9429999999999996,"15min":5.7119999999999997},"max":{"1min":51.045999999999999,"5min":185.47200000000001,"15min":261.80200000000002},"last":10.707000000000001},{"interface":"front","average":{"1min":16.626999999999999,"5min":21.643000000000001,"15min":26.568000000000001},"min":{"1min":8.0180000000000007,"5min":6.3760000000000003,"15min":6.3760000000000003},"max":{"1min":33.390999999999998,"5min":174.38999999999999,"15min":371.35700000000003},"last":16.062999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":23.914999999999999,"5min":27.789999999999999,"15min":27.952999999999999},"min":{"1min":10.460000000000001,"5min":9.4939999999999998,"15min":8.0060000000000002},"max":{"1min":95.795000000000002,"5min":215.31800000000001,"15min":312.44999999999999},"last":13.84},{"interface":"front","average":{"1min":19.725999999999999,"5min":28.399999999999999,"15min":27.664000000000001},"min":{"1min":9.859,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":63.609999999999999,"5min":266.50599999999997,"15min":320.50999999999999},"last":17.959}]},{"osd":7,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":17.331,"5min":26.408000000000001,"15min":28.349},"min":{"1min":9.048,"5min":9.048,"15min":7.9370000000000003},"max":{"1min":55.015000000000001,"5min":193.95400000000001,"15min":319.84199999999998},"last":16.734000000000002},{"interface":"front","average":{"1min":15.836,"5min":26.617000000000001,"15min":27.077000000000002},"min":{"1min":10.515000000000001,"5min":8.9489999999999998,"15min":8.5410000000000004},"max":{"1min":28.527000000000001,"5min":252.232,"15min":315.06900000000002},"last":10.109999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509709,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84352,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287488,"statfs":{"total":96636764160,"available":96550387712,"internally_reserved":0,"allocated":8970240,"data_stored":4460959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.731999999999999,"5min":21.190000000000001,"15min":21.754999999999999},"min":{"1min":4.609,"5min":4.2460000000000004,"15min":4.1159999999999997},"max":{"1min":210.398,"5min":303.58600000000001,"15min":303.58600000000001},"last":13.513999999999999},{"interface":"front","average":{"1min":26.295999999999999,"5min":17.559999999999999,"15min":21.213000000000001},"min":{"1min":6.0049999999999999,"5min":4.8819999999999997,"15min":4.1070000000000002},"max":{"1min":187.82400000000001,"5min":187.82400000000001,"15min":268.68799999999999},"last":9.6069999999999993}]},{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":42.889000000000003,"5min":25.684999999999999,"15min":23.533999999999999},"min":{"1min":7.7990000000000004,"5min":5.6070000000000002,"15min":4.8940000000000001},"max":{"1min":220.96600000000001,"5min":220.96600000000001,"15min":389.303},"last":7.758},{"interface":"front","average":{"1min":43.094999999999999,"5min":25.721,"15min":24.445},"min":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":5.0890000000000004},"max":{"1min":222.328,"5min":222.328,"15min":468.13299999999998},"last":8.7379999999999995}]},{"osd":2,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":19.759,"5min":21.931000000000001,"15min":22.350999999999999},"min":{"1min":8.0250000000000004,"5min":6.8230000000000004,"15min":5.6150000000000002},"max":{"1min":135.97399999999999,"5min":352.09399999999999,"15min":352.09399999999999},"last":11.629},{"interface":"front","average":{"1min":18.385000000000002,"5min":20.425999999999998,"15min":23.646999999999998},"min":{"1min":7.7889999999999997,"5min":6.9850000000000003,"15min":6.9850000000000003},"max":{"1min":94.552999999999997,"5min":184.04599999999999,"15min":287.33699999999999},"last":12.634}]},{"osd":4,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.673000000000002,"5min":25.640000000000001,"15min":28.405000000000001},"min":{"1min":8.6929999999999996,"5min":7.1799999999999997,"15min":6.5010000000000003},"max":{"1min":211.03200000000001,"5min":310.12099999999998,"15min":310.12099999999998},"last":16.564},{"interface":"front","average":{"1min":31.611000000000001,"5min":25.715,"15min":25.268000000000001},"min":{"1min":9.2690000000000001,"5min":6.7169999999999996,"15min":6.0919999999999996},"max":{"1min":187.15700000000001,"5min":359.87799999999999,"15min":359.87799999999999},"last":10.917}]},{"osd":5,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":44.430999999999997,"5min":29.558,"15min":30.042999999999999},"min":{"1min":9.8840000000000003,"5min":9,"15min":6.851},"max":{"1min":274.15300000000002,"5min":360.88099999999997,"15min":564.26700000000005},"last":14.375},{"interface":"front","average":{"1min":50.723999999999997,"5min":28.849,"15min":28.324999999999999},"min":{"1min":10.619,"5min":8.8680000000000003,"15min":8.0109999999999992},"max":{"1min":289.68099999999998,"5min":289.68099999999998,"15min":294.392},"last":20.788}]},{"osd":6,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":30.876999999999999,"5min":29.062999999999999,"15min":30.353999999999999},"min":{"1min":11.016,"5min":9.3369999999999997,"15min":8.8149999999999995},"max":{"1min":197.09200000000001,"5min":374.33999999999997,"15min":374.33999999999997},"last":15.675000000000001},{"interface":"front","average":{"1min":33.301000000000002,"5min":31.440000000000001,"15min":31.552},"min":{"1min":11.416,"5min":11.416,"15min":8.9079999999999995},"max":{"1min":177.63499999999999,"5min":352.79500000000002,"15min":352.79500000000002},"last":17.475000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":36.420000000000002,"5min":32.140999999999998,"15min":34.933},"min":{"1min":12.868,"5min":11.401999999999999,"15min":8.5099999999999998},"max":{"1min":203.70099999999999,"5min":382.68299999999999,"15min":382.68299999999999},"last":19.989999999999998},{"interface":"front","average":{"1min":24.998999999999999,"5min":29.645,"15min":32.134},"min":{"1min":11.976000000000001,"5min":9.0850000000000009,"15min":8.7699999999999996},"max":{"1min":86.024000000000001,"5min":361.47199999999998,"15min":562.05700000000002},"last":18.416}]}]},{"osd":2,"up_from":15,"seq":64424509710,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65920,"kb_used_data":8376,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94305920,"statfs":{"total":96636764160,"available":96569262080,"internally_reserved":0,"allocated":8577024,"data_stored":4196328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:40:02 2023","interfaces":[{"interface":"back","average":{"1min":16.411999999999999,"5min":15.106999999999999,"15min":18.882000000000001},"min":{"1min":4.1559999999999997,"5min":4.1559999999999997,"15min":3.8079999999999998},"max":{"1min":72.049000000000007,"5min":119.291,"15min":337.86700000000002},"last":5.1280000000000001},{"interface":"front","average":{"1min":22.454000000000001,"5min":15.361000000000001,"15min":17.914000000000001},"min":{"1min":6.2690000000000001,"5min":4.4279999999999999,"15min":4.1399999999999997},"max":{"1min":137.864,"5min":137.864,"15min":343.983},"last":11.358000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:39:54 2023","interfaces":[{"interface":"back","average":{"1min":27.044,"5min":18.896999999999998,"15min":19.431999999999999},"min":{"1min":8.4809999999999999,"5min":4.9790000000000001,"15min":4.6559999999999997},"max":{"1min":221.81200000000001,"5min":221.81200000000001,"15min":312.11900000000003},"last":13.146000000000001},{"interface":"front","average":{"1min":24.356000000000002,"5min":19.363,"15min":18.806999999999999},"min":{"1min":6.9180000000000001,"5min":6.9180000000000001,"15min":6.649},"max":{"1min":214.18100000000001,"5min":214.18100000000001,"15min":311.22800000000001},"last":7.4340000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:39:59 2023","interfaces":[{"interface":"back","average":{"1min":19.18,"5min":22.670000000000002,"15min":23.629000000000001},"min":{"1min":7,"5min":7,"15min":6.6379999999999999},"max":{"1min":73.805000000000007,"5min":173.56899999999999,"15min":337.33100000000002},"last":8.1709999999999994},{"interface":"front","average":{"1min":30.245999999999999,"5min":24.972000000000001,"15min":24.486999999999998},"min":{"1min":9.2439999999999998,"5min":7.3620000000000001,"15min":6.6269999999999998},"max":{"1min":196.02099999999999,"5min":197.904,"15min":314.721},"last":14.583}]},{"osd":4,"last update":"Mon Oct 30 12:39:42 2023","interfaces":[{"interface":"back","average":{"1min":16.821999999999999,"5min":22.812000000000001,"15min":23.850999999999999},"min":{"1min":8,"5min":8,"15min":6.415},"max":{"1min":40.204999999999998,"5min":157.77199999999999,"15min":331.834},"last":10.981999999999999},{"interface":"front","average":{"1min":15.441000000000001,"5min":17.044,"15min":18.234999999999999},"min":{"1min":6.8239999999999998,"5min":6.1509999999999998,"15min":5.984},"max":{"1min":47.32,"5min":134.452,"15min":179.89099999999999},"last":13.863}]},{"osd":5,"last update":"Mon Oct 30 12:39:47 2023","interfaces":[{"interface":"back","average":{"1min":19.218,"5min":22.625,"15min":28.466999999999999},"min":{"1min":9.5459999999999994,"5min":8.0619999999999994,"15min":7.4500000000000002},"max":{"1min":42.088000000000001,"5min":201.00299999999999,"15min":326.47699999999998},"last":15.739000000000001},{"interface":"front","average":{"1min":18.844999999999999,"5min":24.981000000000002,"15min":30.864999999999998},"min":{"1min":8.9949999999999992,"5min":8.9949999999999992,"15min":8.9949999999999992},"max":{"1min":40.993000000000002,"5min":238.57300000000001,"15min":338.82100000000003},"last":9.4629999999999992}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":16.959,"5min":19.983000000000001,"15min":20.140000000000001},"min":{"1min":8.1110000000000007,"5min":6.3239999999999998,"15min":6.1100000000000003},"max":{"1min":49.598999999999997,"5min":146.75,"15min":182.97499999999999},"last":11.973000000000001},{"interface":"front","average":{"1min":20.119,"5min":23.864000000000001,"15min":25.939},"min":{"1min":11.311,"5min":9.8819999999999997,"15min":8.7729999999999997},"max":{"1min":45.713000000000001,"5min":154.619,"15min":315.71100000000001},"last":17.225000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:39:54 2023","interfaces":[{"interface":"back","average":{"1min":19.215,"5min":19.186,"15min":19.881},"min":{"1min":8.2750000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"max":{"1min":47.933,"5min":123.926,"15min":228.67400000000001},"last":16.504999999999999},{"interface":"front","average":{"1min":23.645,"5min":25.620000000000001,"15min":26.815000000000001},"min":{"1min":15.757999999999999,"5min":10.462999999999999,"15min":9.0589999999999993},"max":{"1min":39.351999999999997,"5min":160.93600000000001,"15min":328.51799999999997},"last":18.329000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509710,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62728,"kb_used_data":9344,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309112,"statfs":{"total":96636764160,"available":96572530688,"internally_reserved":0,"allocated":9568256,"data_stored":4982001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":10.207000000000001,"5min":11.294,"15min":18.631},"min":{"1min":5.5519999999999996,"5min":4.3760000000000003,"15min":4.3760000000000003},"max":{"1min":24.184999999999999,"5min":75.006,"15min":294.67099999999999},"last":11.714},{"interface":"front","average":{"1min":11.981999999999999,"5min":11.672000000000001,"15min":17.254000000000001},"min":{"1min":6.4210000000000003,"5min":5.2400000000000002,"15min":4.952},"max":{"1min":27.530999999999999,"5min":61.932000000000002,"15min":241.066},"last":9.0540000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":16.41,"5min":14.955,"15min":19.184000000000001},"min":{"1min":6.4299999999999997,"5min":6.4299999999999997,"15min":6.0519999999999996},"max":{"1min":67.140000000000001,"5min":75.491,"15min":295.50599999999997},"last":14.824999999999999},{"interface":"front","average":{"1min":18.949999999999999,"5min":13.51,"15min":19.122},"min":{"1min":6.8810000000000002,"5min":6.1580000000000004,"15min":5.327},"max":{"1min":135.00800000000001,"5min":135.00800000000001,"15min":269.99200000000002},"last":10.198}]},{"osd":3,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":19.390000000000001,"15min":21.367999999999999},"min":{"1min":6.9889999999999999,"5min":6.4779999999999998,"15min":6.4779999999999998},"max":{"1min":25.774999999999999,"5min":310.40800000000002,"15min":551.58000000000004},"last":8.3610000000000007},{"interface":"front","average":{"1min":17.298999999999999,"5min":28.062000000000001,"15min":27.957000000000001},"min":{"1min":9.8819999999999997,"5min":9.8819999999999997,"15min":9.1999999999999993},"max":{"1min":29.579000000000001,"5min":338.88600000000002,"15min":562.06899999999996},"last":12.340999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":14.186,"5min":16.617000000000001,"15min":21.454000000000001},"min":{"1min":7.0469999999999997,"5min":6.7480000000000002,"15min":6.6109999999999998},"max":{"1min":40.465000000000003,"5min":84.212000000000003,"15min":285.26499999999999},"last":13.558999999999999},{"interface":"front","average":{"1min":14.521000000000001,"5min":16.108000000000001,"15min":22.126000000000001},"min":{"1min":9.0500000000000007,"5min":6.7450000000000001,"15min":6.7450000000000001},"max":{"1min":42.087000000000003,"5min":62.616,"15min":327.59699999999998},"last":12.972}]},{"osd":5,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":35.115000000000002,"5min":25.553999999999998,"15min":26.443999999999999},"min":{"1min":9.9410000000000007,"5min":8.1639999999999997,"15min":7.3070000000000004},"max":{"1min":322.59899999999999,"5min":322.59899999999999,"15min":322.59899999999999},"last":15.435},{"interface":"front","average":{"1min":32.095999999999997,"5min":21.616,"15min":24.370999999999999},"min":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"max":{"1min":283.06,"5min":283.06,"15min":286.43799999999999},"last":19.02}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":13.723000000000001,"5min":17.039000000000001,"15min":22.716000000000001},"min":{"1min":7.6059999999999999,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":28.151,"5min":71.662999999999997,"15min":348.91000000000003},"last":10.997999999999999},{"interface":"front","average":{"1min":18.815999999999999,"5min":23.582999999999998,"15min":26.449000000000002},"min":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":8.4819999999999993},"max":{"1min":33.466000000000001,"5min":99.231999999999999,"15min":301.99799999999999},"last":18.256}]},{"osd":7,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":21.850999999999999,"5min":21.949000000000002,"15min":25.800999999999998},"min":{"1min":13.108000000000001,"5min":9.2780000000000005,"15min":7.8739999999999997},"max":{"1min":68.974999999999994,"5min":107.703,"15min":305.19900000000001},"last":16.030999999999999},{"interface":"front","average":{"1min":20.207000000000001,"5min":20.809000000000001,"15min":26.821000000000002},"min":{"1min":8.8640000000000008,"5min":8.8640000000000008,"15min":8.0920000000000005},"max":{"1min":70.406999999999996,"5min":106.678,"15min":277.05200000000002},"last":14.188000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509710,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62148,"kb_used_data":8764,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309692,"statfs":{"total":96636764160,"available":96573124608,"internally_reserved":0,"allocated":8974336,"data_stored":4460964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":14.013,"5min":16.620999999999999,"15min":19.768000000000001},"min":{"1min":6.117,"5min":4.1799999999999997,"15min":4.1799999999999997},"max":{"1min":73.113,"5min":232.798,"15min":287.096},"last":7.9299999999999997},{"interface":"front","average":{"1min":11.015000000000001,"5min":14.513,"15min":22.515999999999998},"min":{"1min":7.6879999999999997,"5min":5.2370000000000001,"15min":4.7590000000000003},"max":{"1min":29.097000000000001,"5min":218.61099999999999,"15min":425.762},"last":14.536}]},{"osd":2,"last update":"Mon Oct 30 12:39:40 2023","interfaces":[{"interface":"back","average":{"1min":21.314,"5min":17.907,"15min":20.555},"min":{"1min":8.3599999999999994,"5min":5.2480000000000002,"15min":5.2480000000000002},"max":{"1min":78.947999999999993,"5min":228.90799999999999,"15min":299.46600000000001},"last":15.111000000000001},{"interface":"front","average":{"1min":28.640000000000001,"5min":18.164000000000001,"15min":20.786000000000001},"min":{"1min":6.7779999999999996,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":113.76000000000001,"5min":138.13499999999999,"15min":258.45400000000001},"last":6.4470000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.376999999999999,"5min":16.640999999999998,"15min":20.597999999999999},"min":{"1min":9.6329999999999991,"5min":6.5499999999999998,"15min":5.2110000000000003},"max":{"1min":41.456000000000003,"5min":166.21600000000001,"15min":264.97199999999998},"last":8.5519999999999996},{"interface":"front","average":{"1min":19.736000000000001,"5min":21.084,"15min":25.367000000000001},"min":{"1min":8.4079999999999995,"5min":8.4079999999999995,"15min":7.218},"max":{"1min":54.859000000000002,"5min":230.73699999999999,"15min":258.93200000000002},"last":16.061}]},{"osd":4,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6620000000000008,"5min":18.376000000000001,"15min":22.655999999999999},"min":{"1min":5.681,"5min":5.681,"15min":5.681},"max":{"1min":24.405999999999999,"5min":209.64599999999999,"15min":209.64599999999999},"last":7.3239999999999998},{"interface":"front","average":{"1min":20.053999999999998,"5min":25.309000000000001,"15min":23.466999999999999},"min":{"1min":8.8819999999999997,"5min":8.0990000000000002,"15min":5.8499999999999996},"max":{"1min":90.787999999999997,"5min":208.92400000000001,"15min":208.92400000000001},"last":13.305}]},{"osd":5,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.98,"5min":29.106000000000002,"15min":29.109999999999999},"min":{"1min":10.965,"5min":8.9440000000000008,"15min":7.5510000000000002},"max":{"1min":31.939,"5min":245.16300000000001,"15min":285.57100000000003},"last":10.73},{"interface":"front","average":{"1min":17.673999999999999,"5min":21.956,"15min":28.129999999999999},"min":{"1min":10.518000000000001,"5min":7.2720000000000002,"15min":7.2720000000000002},"max":{"1min":30.361000000000001,"5min":142.25,"15min":292.07900000000001},"last":16.835000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":15.827999999999999,"5min":20.135999999999999,"15min":21.259},"min":{"1min":7.7000000000000002,"5min":7.4370000000000003,"15min":7.4370000000000003},"max":{"1min":43.283999999999999,"5min":171.99600000000001,"15min":236.40000000000001},"last":9.2919999999999998},{"interface":"front","average":{"1min":18.968,"5min":24.116,"15min":25.454999999999998},"min":{"1min":12.252000000000001,"5min":10.470000000000001,"15min":8.9199999999999999},"max":{"1min":31.385000000000002,"5min":178.59999999999999,"15min":220.68700000000001},"last":11.391999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":19.300999999999998,"5min":20.318000000000001,"15min":24.036000000000001},"min":{"1min":11.853,"5min":9.2100000000000009,"15min":9.2100000000000009},"max":{"1min":32.664999999999999,"5min":143.49600000000001,"15min":245.87100000000001},"last":17.645},{"interface":"front","average":{"1min":16.231000000000002,"5min":22.702000000000002,"15min":24.879999999999999},"min":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":32.033000000000001,"5min":181.15100000000001,"15min":260.04000000000002},"last":9.9879999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575118,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61760,"kb_used_data":8376,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310080,"statfs":{"total":96636764160,"available":96573521920,"internally_reserved":0,"allocated":8577024,"data_stored":4196328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":14.144,"5min":11.411,"15min":20.879000000000001},"min":{"1min":4.5,"5min":4.0960000000000001,"15min":4.0099999999999998},"max":{"1min":110.503,"5min":110.503,"15min":623.43700000000001},"last":7.9260000000000002},{"interface":"front","average":{"1min":10.115,"5min":11.086,"15min":18.956},"min":{"1min":4.9050000000000002,"5min":4.202,"15min":4.202},"max":{"1min":36.860999999999997,"5min":51.253999999999998,"15min":563.08900000000006},"last":10.638}]},{"osd":1,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":16.497,"5min":13.163,"15min":16.117000000000001},"min":{"1min":6.3159999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"max":{"1min":143.386,"5min":143.386,"15min":159.941},"last":7.29},{"interface":"front","average":{"1min":21.492000000000001,"5min":17.748000000000001,"15min":21.466000000000001},"min":{"1min":9.5489999999999995,"5min":7.0679999999999996,"15min":7.008},"max":{"1min":153.08199999999999,"5min":153.08199999999999,"15min":266.30599999999998},"last":11.864000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:39:51 2023","interfaces":[{"interface":"back","average":{"1min":22.289999999999999,"5min":20.649000000000001,"15min":20.420999999999999},"min":{"1min":6.5949999999999998,"5min":6.1749999999999998,"15min":5.0529999999999999},"max":{"1min":190.024,"5min":329.81999999999999,"15min":329.81999999999999},"last":8.5380000000000003},{"interface":"front","average":{"1min":14.016,"5min":16.754000000000001,"15min":20.099},"min":{"1min":7.9550000000000001,"5min":6.8129999999999997,"15min":6.0350000000000001},"max":{"1min":51.386000000000003,"5min":132.25800000000001,"15min":132.25800000000001},"last":16.227}]},{"osd":3,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":20.202999999999999,"5min":19.655999999999999,"15min":23.643000000000001},"min":{"1min":8.0839999999999996,"5min":6.4950000000000001,"15min":6.4950000000000001},"max":{"1min":124.31399999999999,"5min":124.31399999999999,"15min":185.98400000000001},"last":17.155999999999999},{"interface":"front","average":{"1min":22.238,"5min":22.606999999999999,"15min":24.629999999999999},"min":{"1min":8.4100000000000001,"5min":7.7000000000000002,"15min":6.8159999999999998},"max":{"1min":159.69999999999999,"5min":247.65899999999999,"15min":247.65899999999999},"last":9.2769999999999992}]},{"osd":4,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":17.640000000000001,"5min":19.411000000000001,"15min":24.236999999999998},"min":{"1min":9.4979999999999993,"5min":5.8150000000000004,"15min":5.8150000000000004},"max":{"1min":50.786000000000001,"5min":117.967,"15min":232.17099999999999},"last":15.43},{"interface":"front","average":{"1min":18.247,"5min":18.09,"15min":22.715},"min":{"1min":6.806,"5min":6.806,"15min":6.6280000000000001},"max":{"1min":100.952,"5min":100.952,"15min":267.11000000000001},"last":10.025}]},{"osd":5,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":20.390000000000001,"5min":22.111000000000001,"15min":27.120999999999999},"min":{"1min":7.6349999999999998,"5min":7.1440000000000001,"15min":6.5110000000000001},"max":{"1min":125.926,"5min":186.54499999999999,"15min":203.66399999999999},"last":13.166},{"interface":"front","average":{"1min":16.417999999999999,"5min":23.021999999999998,"15min":27.204000000000001},"min":{"1min":8.6980000000000004,"5min":8.6980000000000004,"15min":7.1500000000000004},"max":{"1min":46.524000000000001,"5min":185.61099999999999,"15min":275.88099999999997},"last":11.302}]},{"osd":7,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":18.094000000000001,"5min":20.02,"15min":26.045000000000002},"min":{"1min":8.952,"5min":8.952,"15min":7.5579999999999998},"max":{"1min":50.177999999999997,"5min":99.480999999999995,"15min":227.18600000000001},"last":17.957000000000001},{"interface":"front","average":{"1min":17.981000000000002,"5min":21.117000000000001,"15min":25.186},"min":{"1min":9.5609999999999999,"5min":8.8019999999999996,"15min":8.8019999999999996},"max":{"1min":51.954000000000001,"5min":108.705,"15min":264.99299999999999},"last":12.49}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:40:15.603 INFO:tasks.ceph:pgid 20.c last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:40:15.603 INFO:tasks.ceph:pgid 20.13 last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:40:15.604 INFO:tasks.ceph:pgid 20.1f last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:40:15.604 INFO:tasks.ceph:pgid 20.1d last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:40:15.604 INFO:tasks.ceph:pgid 20.1c last_scrub_stamp 2023-10-30T12:19:17.767568+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=19, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=39, tm_sec=46, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:40:15.605 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:40:17.480 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:32.554 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:17.482 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:32.556 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:17.498 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:32.572 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:17.523 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:32.570 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:17.527 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:32.574 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:17.531 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:32.604 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:17.532 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:32.606 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:17.540 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:32.587 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:17.545 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:32.618 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:22.491 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:37.564 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:22.495 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:37.569 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:22.510 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:37.584 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:22.539 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:37.587 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:22.542 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:37.615 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:22.543 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:37.590 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:22.545 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:37.618 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:22.553 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:37.601 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:22.555 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:37.629 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:35.605 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:40:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.707+0000 7f09b0ebb640 10 monclient: get_monmap_and_config 2023-10-30T12:40:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.707+0000 7f09b0ebb640 10 monclient: build_initial_monmap 2023-10-30T12:40:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.708+0000 7f09b0ebb640 10 monclient: monmap: 2023-10-30T12:40:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:40:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:40:35.709043+0000 2023-10-30T12:40:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:40:35.709043+0000 2023-10-30T12:40:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:40:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:40:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:40:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:40:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:40:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:40:35.712 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.708+0000 7f09b0ebb640 1 Processor -- start 2023-10-30T12:40:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.708+0000 7f09b0ebb640 1 -- start start 2023-10-30T12:40:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.708+0000 7f09b0ebb640 10 monclient: init 2023-10-30T12:40:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 10 monclient: _reopen_session rank -1 2023-10-30T12:40:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T12:40:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09ac0b7c10 addr v1:172.21.15.148:6789/0 2023-10-30T12:40:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09ac141940 addr v1:172.21.15.150:6789/0 2023-10-30T12:40:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f09ac14a460 0x7f09ac14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:35.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 10 monclient(hunting): picked mon.noname-b con 0x7f09ac14a460 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:40:35.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac0a9350 con 0x7f09ac0b7c10 2023-10-30T12:40:35.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac0aa6a0 con 0x7f09ac141940 2023-10-30T12:40:35.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:35.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0670a0 con 0x7f09ac14a460 2023-10-30T12:40:35.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 10 monclient(hunting): _renew_subs 2023-10-30T12:40:35.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09b0ebb640 10 monclient(hunting): authenticate will time out at 2927.914340s 2023-10-30T12:40:35.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09aa59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f09ac0b7c10 0x7f09ac0a8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45960/0 (socket says 172.21.15.148:45960) 2023-10-30T12:40:35.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.709+0000 7f09aa59c640 1 -- 172.21.15.148:0/2998733672 learned_addr learned my addr 172.21.15.148:0/2998733672 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:40:35.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088452826 0 0) 0x7f09a4001110 con 0x7f09ac0b7c10 2023-10-30T12:40:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 10 monclient(hunting): _negotiate 2023-10-30T12:40:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 10 monclient(hunting): my global_id is 16329 2023-10-30T12:40:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0994003520 con 0x7f09ac0b7c10 2023-10-30T12:40:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653680414 0 0) 0x7f09a0001110 con 0x7f09ac141940 2023-10-30T12:40:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 10 monclient(hunting): _negotiate 2023-10-30T12:40:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 10 monclient(hunting): my global_id is 22153 2023-10-30T12:40:35.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.712+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0994003840 con 0x7f09ac141940 2023-10-30T12:40:35.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 805415030 0 0) 0x7f09a4001510 con 0x7f09ac0b7c10 2023-10-30T12:40:35.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f09ac14a460 msgr2=0x7f09ac14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:35.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 --2- 172.21.15.148:0/2998733672 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f09ac14a460 0x7f09ac14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:35.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 >> v1:172.21.15.150:6789/0 conn(0x7f09ac141940 legacy=0x7f09ac141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:35.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient: _finish_hunting 0 2023-10-30T12:40:35.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:40:35.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:40:35.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.148:6789/0 2023-10-30T12:40:35.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac0aa910 con 0x7f09ac0b7c10 2023-10-30T12:40:35.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient: _finish_auth 0 2023-10-30T12:40:35.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient: _check_auth_tickets 2023-10-30T12:40:35.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:40:35.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1567732308 0 0) 0x7f09a40019b0 con 0x7f09ac0b7c10 2023-10-30T12:40:35.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:35.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:40:35.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient: dump: 2023-10-30T12:40:35.726 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:35.726 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:35.726 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:35.726 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:35.726 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:35.727 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] but i'm connected to v1:172.21.15.148:6789/0, reconnecting 2023-10-30T12:40:35.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient: _reopen_session rank -1 2023-10-30T12:40:35.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 >> v1:172.21.15.148:6789/0 conn(0x7f09ac0b7c10 legacy=0x7f09ac0a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:35.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:40:35.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 --2- 172.21.15.148:0/2998733672 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f09ac14a460 0x7f0994003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:35.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient(hunting): picked mon.c con 0x7f09ac14a460 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:40:35.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 --2- 172.21.15.148:0/2998733672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0994004270 0x7f09940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:35.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient(hunting): picked mon.a con 0x7f0994004270 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:40:35.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 --2- 172.21.15.148:0/2998733672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0994004f50 0x7f099400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:35.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient(hunting): picked mon.b con 0x7f0994004f50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:40:35.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:35.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0670a0 con 0x7f0994004270 2023-10-30T12:40:35.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:35.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099400bec0 con 0x7f0994004f50 2023-10-30T12:40:35.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:35.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099400c040 con 0x7f09ac14a460 2023-10-30T12:40:35.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient(hunting): _renew_subs 2023-10-30T12:40:35.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.715+0000 7f09a959a640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:40:35.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.717+0000 7f09aad9d640 1 --2- 172.21.15.148:0/2998733672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0994004f50 0x7f099400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:35.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09aad9d640 10 monclient(hunting): get_auth_request con 0x7f0994004f50 auth_method 0 2023-10-30T12:40:35.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:35.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09aad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:35.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09aad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:40:35.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09a9d9b640 1 --2- 172.21.15.148:0/2998733672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0994004270 0x7f09940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:35.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:35.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:35.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:40:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09a9d9b640 10 monclient(hunting): get_auth_request con 0x7f0994004270 auth_method 0 2023-10-30T12:40:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09a9d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.718+0000 7f09a9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:40:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.719+0000 7f09a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.719+0000 7f09a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.719+0000 7f09a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:40:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 10 monclient(hunting): handle_auth_done global_id 16329 payload 293 2023-10-30T12:40:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 1 -- 172.21.15.148:0/2998733672 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f09ac14a460 msgr2=0x7f0994003cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 1 --2- 172.21.15.148:0/2998733672 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f09ac14a460 0x7f0994003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 1 -- 172.21.15.148:0/2998733672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0994004270 msgr2=0x7f09940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:35.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 1 --2- 172.21.15.148:0/2998733672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0994004270 0x7f09940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:35.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:40:35.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 1 monclient: found mon.b 2023-10-30T12:40:35.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:40:35.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:40:35.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 1 -- 172.21.15.148:0/2998733672 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4001510 con 0x7f0994004f50 2023-10-30T12:40:35.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 10 monclient: _finish_auth 0 2023-10-30T12:40:35.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 10 monclient: _check_auth_tickets 2023-10-30T12:40:35.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09aad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:40:35.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09b0ebb640 5 monclient: authenticate success, global_id 16329 2023-10-30T12:40:35.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.720+0000 7f09b0ebb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:40:35.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.723+0000 7f09aad9d640 1 --2- 172.21.15.148:0/2998733672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0994004f50 0x7f099400b980 secure :-1 s=READY pgs=5459 cs=0 l=1 rev1=1 crypto rx=0x7f099c009e10 tx=0x7f09940057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=916f0ad6dca787df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:35.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.726+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c02cd50 con 0x7f0994004f50 2023-10-30T12:40:35.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:35.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09a959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:40:35.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09a959a640 10 monclient: dump: 2023-10-30T12:40:35.742 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:35.742 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:35.742 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:35.742 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:35.743 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:35.743 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:35.743 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:35.743 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:35.743 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:35.743 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:35.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c034440 con 0x7f0994004f50 2023-10-30T12:40:35.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:40:35.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09a959a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:40:35.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 10 monclient: get_monmap_and_config success 2023-10-30T12:40:35.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 4 set_mon_vals no callback set 2023-10-30T12:40:35.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09a959a640 1 -- 172.21.15.148:0/2998733672 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c034840 con 0x7f0994004f50 2023-10-30T12:40:35.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:35.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09a959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:40:35.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09a959a640 10 monclient: dump: 2023-10-30T12:40:35.745 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:35.746 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:35.746 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:35.746 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:35.746 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:35.746 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:35.746 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:35.747 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:35.747 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:35.747 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:35.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 10 monclient: shutdown 2023-10-30T12:40:35.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:40:35.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 1 -- 172.21.15.148:0/2998733672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0994004f50 msgr2=0x7f099400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:35.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 1 --2- 172.21.15.148:0/2998733672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0994004f50 0x7f099400b980 secure :-1 s=READY pgs=5459 cs=0 l=1 rev1=1 crypto rx=0x7f099c009e10 tx=0x7f09940057f0 comp rx=0 tx=0).stop 2023-10-30T12:40:35.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09aad9d640 1 -- 172.21.15.148:0/2998733672 reap_dead start 2023-10-30T12:40:35.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 1 -- 172.21.15.148:0/2998733672 shutdown_connections 2023-10-30T12:40:35.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 1 -- 172.21.15.148:0/2998733672 >> 172.21.15.148:0/2998733672 conn(0x7f09ac06c660 msgr2=0x7f09ac0b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:35.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 1 -- 172.21.15.148:0/2998733672 shutdown_connections 2023-10-30T12:40:35.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.727+0000 7f09b0ebb640 1 -- 172.21.15.148:0/2998733672 wait complete. 2023-10-30T12:40:35.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.728+0000 7f09b0ebb640 10 monclient: build_initial_monmap 2023-10-30T12:40:35.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.728+0000 7f09b0ebb640 10 monclient: monmap: 2023-10-30T12:40:35.749 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:40:35.749 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:40:35.750 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:40:35.750 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:40:35.750 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:40:35.750 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:40:35.750 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:40:35.751 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:40:35.751 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:40:35.751 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:35.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.728+0000 7f09b0ebb640 1 librados: starting msgr at 2023-10-30T12:40:35.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.728+0000 7f09b0ebb640 1 librados: starting objecter 2023-10-30T12:40:35.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 1 Processor -- start 2023-10-30T12:40:35.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 1 -- start start 2023-10-30T12:40:35.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 1 librados: setting wanted keys 2023-10-30T12:40:35.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 1 librados: calling monclient init 2023-10-30T12:40:35.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient: init 2023-10-30T12:40:35.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient: _reopen_session rank -1 2023-10-30T12:40:35.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:40:35.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0994004f50 0x7f09ac0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:35.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient(hunting): picked mon.noname-c con 0x7f0994004f50 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:40:35.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0ac750 0x7f09ac136400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:35.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient(hunting): picked mon.noname-a con 0x7f09ac0ac750 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:40:35.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac136940 0x7f09ac1d9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:35.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient(hunting): picked mon.noname-b con 0x7f09ac136940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:40:35.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:35.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0b3400 con 0x7f09ac0ac750 2023-10-30T12:40:35.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:35.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0acc70 con 0x7f09ac136940 2023-10-30T12:40:35.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:35.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0ad3f0 con 0x7f0994004f50 2023-10-30T12:40:35.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient(hunting): _renew_subs 2023-10-30T12:40:35.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.729+0000 7f09b0ebb640 10 monclient(hunting): authenticate will time out at 2927.934545s 2023-10-30T12:40:35.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09a9d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0ac750 0x7f09ac136400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:35.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09aad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac136940 0x7f09ac1d9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:35.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09a9d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0ac750 0x7f09ac136400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43762/0 (socket says 172.21.15.148:43762) 2023-10-30T12:40:35.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09a9d9b640 1 -- 172.21.15.148:0/1782799232 learned_addr learned my addr 172.21.15.148:0/1782799232 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:40:35.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09a9d9b640 10 monclient(hunting): get_auth_request con 0x7f09ac0ac750 auth_method 0 2023-10-30T12:40:35.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:35.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09a9d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:35.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09a9d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:35.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09aad9d640 10 monclient(hunting): get_auth_request con 0x7f09ac136940 auth_method 0 2023-10-30T12:40:35.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:35.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09aad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:35.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09aad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:35.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:35.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:35.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:35.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:35.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.730+0000 7f09aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:35.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.731+0000 7f09aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:35.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09aad9d640 10 monclient(hunting): handle_auth_done global_id 22156 payload 824 2023-10-30T12:40:35.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09aad9d640 1 -- 172.21.15.148:0/1782799232 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0994004f50 msgr2=0x7f09ac0b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:35.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09aad9d640 1 --2- 172.21.15.148:0/1782799232 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0994004f50 0x7f09ac0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:35.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09aad9d640 1 -- 172.21.15.148:0/1782799232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0ac750 msgr2=0x7f09ac136400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:35.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09aad9d640 1 --2- 172.21.15.148:0/1782799232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0ac750 0x7f09ac136400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:35.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09aad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:40:35.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09aad9d640 1 monclient: found mon.noname-b 2023-10-30T12:40:35.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:40:35.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09aad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2023-10-30T12:40:35.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09aad9d640 1 -- 172.21.15.148:0/1782799232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac136dc0 con 0x7f09ac136940 2023-10-30T12:40:35.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.733+0000 7f09a9d9b640 1 --2- 172.21.15.148:0/1782799232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0ac750 0x7f09ac136400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:40:35.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.736+0000 7f09aad9d640 1 --2- 172.21.15.148:0/1782799232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac136940 0x7f09ac1d9f90 secure :-1 s=READY pgs=5460 cs=0 l=1 rev1=1 crypto rx=0x7f099c033000 tx=0x7f099c004b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:35.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.736+0000 7f099affd640 10 client.?.objecter ms_handle_connect 0x7f09ac136940 2023-10-30T12:40:35.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.736+0000 7f099affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:40:35.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.739+0000 7f099affd640 1 -- 172.21.15.148:0/1782799232 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c034db0 con 0x7f09ac136940 2023-10-30T12:40:35.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:35.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:40:35.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 10 monclient: dump: 2023-10-30T12:40:35.763 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:35.764 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:35.764 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:35.764 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:35.764 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:35.764 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:35.765 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:35.765 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:35.765 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:35.765 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:35.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 10 monclient: _finish_auth 0 2023-10-30T12:40:35.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 10 monclient: _check_auth_tickets 2023-10-30T12:40:35.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:40:35.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 1 -- 172.21.15.148:0/1782799232 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c03f420 con 0x7f09ac136940 2023-10-30T12:40:35.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:40:35.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 1 -- 172.21.15.148:0/1782799232 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c03f7e0 con 0x7f09ac136940 2023-10-30T12:40:35.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f09b0ebb640 5 monclient: authenticate success, global_id 22156 2023-10-30T12:40:35.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f09b0ebb640 10 monclient: _renew_subs 2023-10-30T12:40:35.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f09b0ebb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:40:35.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f09b0ebb640 1 -- 172.21.15.148:0/1782799232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ac0b6910 con 0x7f09ac136940 2023-10-30T12:40:35.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f09a8d99640 4 set_mon_vals no callback set 2023-10-30T12:40:35.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:40:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f099affd640 10 monclient: dump: 2023-10-30T12:40:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:35.770 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:35.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f09b0ebb640 10 client.22156.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:40:35.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f09b0ebb640 10 monclient: _renew_subs 2023-10-30T12:40:35.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f09b0ebb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:40:35.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f09b0ebb640 1 -- 172.21.15.148:0/1782799232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac1da4d0 con 0x7f09ac136940 2023-10-30T12:40:35.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.740+0000 7f09b0ebb640 1 librados: init done 2023-10-30T12:40:35.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.741+0000 7f09b0ebb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:40:35.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.741+0000 7f09b0ebb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:40:35.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.741+0000 7f09b0ebb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2023-10-30T12:40:35.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.741+0000 7f09b0ebb640 1 -- 172.21.15.148:0/1782799232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac075550 con 0x7f09ac136940 2023-10-30T12:40:35.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.982+0000 7f099affd640 1 -- 172.21.15.148:0/1782799232 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f099c03f950 con 0x7f09ac136940 2023-10-30T12:40:35.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.982+0000 7f099affd640 1 --2- 172.21.15.148:0/1782799232 >> v2:172.21.15.148:6800/3205090179 conn(0x7f098c06c810 0x7f098c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:35.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.982+0000 7f09aa59c640 1 --2- 172.21.15.148:0/1782799232 >> v2:172.21.15.148:6800/3205090179 conn(0x7f098c06c810 0x7f098c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:35.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.982+0000 7f09aa59c640 10 monclient: get_auth_request con 0x7f098c06c810 auth_method 0 2023-10-30T12:40:35.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.982+0000 7f09aa59c640 1 --2- 172.21.15.148:0/1782799232 >> v2:172.21.15.148:6800/3205090179 conn(0x7f098c06c810 0x7f098c06ecd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f09a40078d0 tx=0x7f09a4001680 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:35.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.982+0000 7f099affd640 10 client.22156.objecter ms_handle_connect 0x7f098c06c810 2023-10-30T12:40:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.987+0000 7f099affd640 1 -- 172.21.15.148:0/1782799232 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(900..900 src has 739..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f099c07a1f0 con 0x7f09ac136940 2023-10-30T12:40:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.987+0000 7f099affd640 10 client.22156.objecter ms_dispatch 0x7f09ac000b90 osd_map(900..900 src has 739..900) v4 2023-10-30T12:40:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.987+0000 7f099affd640 3 client.22156.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:40:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.987+0000 7f099affd640 3 client.22156.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:40:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.987+0000 7f099affd640 20 client.22156.objecter dump_active .. 0 homeless 2023-10-30T12:40:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.997+0000 7f099affd640 1 -- 172.21.15.148:0/1782799232 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f099c006e50 con 0x7f09ac136940 2023-10-30T12:40:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.997+0000 7f099affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:40:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:35.997+0000 7f099affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:40:36.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.164+0000 7f09b0ebb640 1 -- 172.21.15.148:0/1782799232 --> v2:172.21.15.148:6800/3205090179 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f09ac0ac090 con 0x7f098c06c810 2023-10-30T12:40:36.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.168+0000 7f099affd640 1 -- 172.21.15.148:0/1782799232 <== mgr.4104 v2:172.21.15.148:6800/3205090179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152366 (secure 0 0 0) 0x7f09ac0ac090 con 0x7f098c06c810 2023-10-30T12:40:36.171 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T12:40:36.172 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T12:40:36.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.170+0000 7f09b0ebb640 10 librados: watch_flush enter 2023-10-30T12:40:36.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.170+0000 7f09b0ebb640 10 librados: watch_flush exit 2023-10-30T12:40:36.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.170+0000 7f09b0ebb640 20 client.22156.objecter shutdown clearing up homeless session... 2023-10-30T12:40:36.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.170+0000 7f09b0ebb640 10 client.22156.objecter successfully canceled tick 2023-10-30T12:40:36.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 -- 172.21.15.148:0/1782799232 >> v2:172.21.15.148:6800/3205090179 conn(0x7f098c06c810 msgr2=0x7f098c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 --2- 172.21.15.148:0/1782799232 >> v2:172.21.15.148:6800/3205090179 conn(0x7f098c06c810 0x7f098c06ecd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f09a40078d0 tx=0x7f09a4001680 comp rx=0 tx=0).stop 2023-10-30T12:40:36.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 10 monclient: shutdown 2023-10-30T12:40:36.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:40:36.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 -- 172.21.15.148:0/1782799232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac136940 msgr2=0x7f09ac1d9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 --2- 172.21.15.148:0/1782799232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac136940 0x7f09ac1d9f90 secure :-1 s=READY pgs=5460 cs=0 l=1 rev1=1 crypto rx=0x7f099c033000 tx=0x7f099c004b50 comp rx=0 tx=0).stop 2023-10-30T12:40:36.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 -- 172.21.15.148:0/1782799232 shutdown_connections 2023-10-30T12:40:36.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 --2- 172.21.15.148:0/1782799232 >> v2:172.21.15.148:6800/3205090179 conn(0x7f098c06c810 0x7f098c06ecd0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 --2- 172.21.15.148:0/1782799232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac136940 0x7f09ac1d9f90 unknown :-1 s=CLOSED pgs=5460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 --2- 172.21.15.148:0/1782799232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0ac750 0x7f09ac136400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 --2- 172.21.15.148:0/1782799232 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0994004f50 0x7f09ac0b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 -- 172.21.15.148:0/1782799232 >> 172.21.15.148:0/1782799232 conn(0x7f09ac06c660 msgr2=0x7f09ac13a810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:36.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 -- 172.21.15.148:0/1782799232 shutdown_connections 2023-10-30T12:40:36.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 -- 172.21.15.148:0/1782799232 wait complete. 2023-10-30T12:40:36.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.171+0000 7f09b0ebb640 1 librados: shutdown 2023-10-30T12:40:36.184 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2023-10-30T12:40:35.321129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":541976,"kb_used_data":70488,"kb_used_omap":12,"kb_used_meta":471411,"kb_avail":754432744,"statfs":{"total":773094113280,"available":772539129856,"internally_reserved":0,"allocated":72179712,"data_stored":36200504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":482725433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002584"},"pg_stats":[{"pgid":"20.8","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:59.863155+0000","last_change":"2023-10-30T12:39:59.864177+0000","last_active":"2023-10-30T12:39:59.863155+0000","last_peered":"2023-10-30T12:39:59.863155+0000","last_clean":"2023-10-30T12:39:59.863155+0000","last_became_active":"2023-10-30T12:19:35.850666+0000","last_became_peered":"2023-10-30T12:19:35.850666+0000","last_unstale":"2023-10-30T12:39:59.863155+0000","last_undegraded":"2023-10-30T12:39:59.863155+0000","last_fullsized":"2023-10-30T12:39:59.863155+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:59.861807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:59.861807+0000","last_clean_scrub_stamp":"2023-10-30T12:39:59.861807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:02.439914+0000","scrub_duration":0.027726994000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:53.854380+0000","last_change":"2023-10-30T12:39:53.859778+0000","last_active":"2023-10-30T12:39:53.854380+0000","last_peered":"2023-10-30T12:39:53.854380+0000","last_clean":"2023-10-30T12:39:53.854380+0000","last_became_active":"2023-10-30T12:19:38.546595+0000","last_became_peered":"2023-10-30T12:19:38.546595+0000","last_unstale":"2023-10-30T12:39:53.854380+0000","last_undegraded":"2023-10-30T12:39:53.854380+0000","last_fullsized":"2023-10-30T12:39:53.854380+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:53.852466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:53.852466+0000","last_clean_scrub_stamp":"2023-10-30T12:39:53.852466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:59.053056+0000","scrub_duration":0.045416113000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:02.886563+0000","last_change":"2023-10-30T12:40:02.887616+0000","last_active":"2023-10-30T12:40:02.886563+0000","last_peered":"2023-10-30T12:40:02.886563+0000","last_clean":"2023-10-30T12:40:02.886563+0000","last_became_active":"2023-10-30T12:19:35.871000+0000","last_became_peered":"2023-10-30T12:19:35.871000+0000","last_unstale":"2023-10-30T12:40:02.886563+0000","last_undegraded":"2023-10-30T12:40:02.886563+0000","last_fullsized":"2023-10-30T12:40:02.886563+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:02.885224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:02.885224+0000","last_clean_scrub_stamp":"2023-10-30T12:40:02.885224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:15.171615+0000","scrub_duration":0.032834037000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":854,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:59.584800+0000","last_change":"2023-10-30T12:39:59.585771+0000","last_active":"2023-10-30T12:39:59.584800+0000","last_peered":"2023-10-30T12:39:59.584800+0000","last_clean":"2023-10-30T12:39:59.584800+0000","last_became_active":"2023-10-30T12:19:29.594505+0000","last_became_peered":"2023-10-30T12:19:29.594505+0000","last_unstale":"2023-10-30T12:39:59.584800+0000","last_undegraded":"2023-10-30T12:39:59.584800+0000","last_fullsized":"2023-10-30T12:39:59.584800+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:59.583409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:59.583409+0000","last_clean_scrub_stamp":"2023-10-30T12:39:59.583409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:47.141085+0000","scrub_duration":0.076237307000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":858,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:15.167411+0000","last_change":"2023-10-30T12:40:15.168410+0000","last_active":"2023-10-30T12:40:15.167411+0000","last_peered":"2023-10-30T12:40:15.167411+0000","last_clean":"2023-10-30T12:40:15.167411+0000","last_became_active":"2023-10-30T12:19:27.515739+0000","last_became_peered":"2023-10-30T12:19:27.515739+0000","last_unstale":"2023-10-30T12:40:15.167411+0000","last_undegraded":"2023-10-30T12:40:15.167411+0000","last_fullsized":"2023-10-30T12:40:15.167411+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:15.166029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:15.166029+0000","last_clean_scrub_stamp":"2023-10-30T12:40:15.166029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:26:22.258867+0000","scrub_duration":0.040030639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":847,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:07.070717+0000","last_change":"2023-10-30T12:40:07.071822+0000","last_active":"2023-10-30T12:40:07.070717+0000","last_peered":"2023-10-30T12:40:07.070717+0000","last_clean":"2023-10-30T12:40:07.070717+0000","last_became_active":"2023-10-30T12:19:31.894071+0000","last_became_peered":"2023-10-30T12:19:31.894071+0000","last_unstale":"2023-10-30T12:40:07.070717+0000","last_undegraded":"2023-10-30T12:40:07.070717+0000","last_fullsized":"2023-10-30T12:40:07.070717+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:07.069366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:07.069366+0000","last_clean_scrub_stamp":"2023-10-30T12:40:07.069366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:49.555106+0000","scrub_duration":0.059461060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:03.248922+0000","last_change":"2023-10-30T12:40:03.249927+0000","last_active":"2023-10-30T12:40:03.248922+0000","last_peered":"2023-10-30T12:40:03.248922+0000","last_clean":"2023-10-30T12:40:03.248922+0000","last_became_active":"2023-10-30T12:19:28.142591+0000","last_became_peered":"2023-10-30T12:19:28.142591+0000","last_unstale":"2023-10-30T12:40:03.248922+0000","last_undegraded":"2023-10-30T12:40:03.248922+0000","last_fullsized":"2023-10-30T12:40:03.248922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:03.247585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:03.247585+0000","last_clean_scrub_stamp":"2023-10-30T12:40:03.247585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:24.330050+0000","scrub_duration":0.032344831999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:58.933952+0000","last_change":"2023-10-30T12:39:58.934987+0000","last_active":"2023-10-30T12:39:58.933952+0000","last_peered":"2023-10-30T12:39:58.933952+0000","last_clean":"2023-10-30T12:39:58.933952+0000","last_became_active":"2023-10-30T12:19:38.643834+0000","last_became_peered":"2023-10-30T12:19:38.643834+0000","last_unstale":"2023-10-30T12:39:58.933952+0000","last_undegraded":"2023-10-30T12:39:58.933952+0000","last_fullsized":"2023-10-30T12:39:58.933952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:58.932666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:58.932666+0000","last_clean_scrub_stamp":"2023-10-30T12:39:58.932666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:07.378282+0000","scrub_duration":0.063031205000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:57.536855+0000","last_change":"2023-10-30T12:39:57.537905+0000","last_active":"2023-10-30T12:39:57.536855+0000","last_peered":"2023-10-30T12:39:57.536855+0000","last_clean":"2023-10-30T12:39:57.536855+0000","last_became_active":"2023-10-30T12:19:29.492043+0000","last_became_peered":"2023-10-30T12:19:29.492043+0000","last_unstale":"2023-10-30T12:39:57.536855+0000","last_undegraded":"2023-10-30T12:39:57.536855+0000","last_fullsized":"2023-10-30T12:39:57.536855+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:57.535422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:57.535422+0000","last_clean_scrub_stamp":"2023-10-30T12:39:57.535422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:15:16.187111+0000","scrub_duration":0.050567743999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1","version":"900'8","reported_seq":854,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:56.071751+0000","last_change":"2023-10-30T12:39:56.071751+0000","last_active":"2023-10-30T12:39:56.071751+0000","last_peered":"2023-10-30T12:39:56.071751+0000","last_clean":"2023-10-30T12:39:56.071751+0000","last_became_active":"2023-10-30T12:19:25.212397+0000","last_became_peered":"2023-10-30T12:19:25.212397+0000","last_unstale":"2023-10-30T12:39:56.071751+0000","last_undegraded":"2023-10-30T12:39:56.071751+0000","last_fullsized":"2023-10-30T12:39:56.071751+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'8","last_scrub_stamp":"2023-10-30T12:39:56.069074+0000","last_deep_scrub":"900'8","last_deep_scrub_stamp":"2023-10-30T12:39:56.069074+0000","last_clean_scrub_stamp":"2023-10-30T12:39:56.069074+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:01.898778+0000","scrub_duration":0.079575325000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":859,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:10.739967+0000","last_change":"2023-10-30T12:40:10.739967+0000","last_active":"2023-10-30T12:40:10.739967+0000","last_peered":"2023-10-30T12:40:10.739967+0000","last_clean":"2023-10-30T12:40:10.739967+0000","last_became_active":"2023-10-30T12:19:29.491706+0000","last_became_peered":"2023-10-30T12:19:29.491706+0000","last_unstale":"2023-10-30T12:40:10.739967+0000","last_undegraded":"2023-10-30T12:40:10.739967+0000","last_fullsized":"2023-10-30T12:40:10.739967+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:10.737405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:10.737405+0000","last_clean_scrub_stamp":"2023-10-30T12:40:10.737405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:27.660864+0000","scrub_duration":0.029906892000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":860,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:02.130087+0000","last_change":"2023-10-30T12:40:02.130087+0000","last_active":"2023-10-30T12:40:02.130087+0000","last_peered":"2023-10-30T12:40:02.130087+0000","last_clean":"2023-10-30T12:40:02.130087+0000","last_became_active":"2023-10-30T12:19:32.988808+0000","last_became_peered":"2023-10-30T12:19:32.988808+0000","last_unstale":"2023-10-30T12:40:02.130087+0000","last_undegraded":"2023-10-30T12:40:02.130087+0000","last_fullsized":"2023-10-30T12:40:02.130087+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:02.127694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:02.127694+0000","last_clean_scrub_stamp":"2023-10-30T12:40:02.127694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:56.157165+0000","scrub_duration":0.067790453000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":859,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:07.139048+0000","last_change":"2023-10-30T12:40:07.139048+0000","last_active":"2023-10-30T12:40:07.139048+0000","last_peered":"2023-10-30T12:40:07.139048+0000","last_clean":"2023-10-30T12:40:07.139048+0000","last_became_active":"2023-10-30T12:19:28.916380+0000","last_became_peered":"2023-10-30T12:19:28.916380+0000","last_unstale":"2023-10-30T12:40:07.139048+0000","last_undegraded":"2023-10-30T12:40:07.139048+0000","last_fullsized":"2023-10-30T12:40:07.139048+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:07.136266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:07.136266+0000","last_clean_scrub_stamp":"2023-10-30T12:40:07.136266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:22.811859+0000","scrub_duration":0.063096992000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":864,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:59.550234+0000","last_change":"2023-10-30T12:39:59.551230+0000","last_active":"2023-10-30T12:39:59.550234+0000","last_peered":"2023-10-30T12:39:59.550234+0000","last_clean":"2023-10-30T12:39:59.550234+0000","last_became_active":"2023-10-30T12:18:01.568787+0000","last_became_peered":"2023-10-30T12:18:01.568787+0000","last_unstale":"2023-10-30T12:39:59.550234+0000","last_undegraded":"2023-10-30T12:39:59.550234+0000","last_fullsized":"2023-10-30T12:39:59.550234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:59.548935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:59.548935+0000","last_clean_scrub_stamp":"2023-10-30T12:39:59.548935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:50:31.515390+0000","scrub_duration":0.034566703999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":847,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:53.220177+0000","last_change":"2023-10-30T12:39:53.220177+0000","last_active":"2023-10-30T12:39:53.220177+0000","last_peered":"2023-10-30T12:39:53.220177+0000","last_clean":"2023-10-30T12:39:53.220177+0000","last_became_active":"2023-10-30T12:19:40.940463+0000","last_became_peered":"2023-10-30T12:19:40.940463+0000","last_unstale":"2023-10-30T12:39:53.220177+0000","last_undegraded":"2023-10-30T12:39:53.220177+0000","last_fullsized":"2023-10-30T12:39:53.220177+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:53.217887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:53.217887+0000","last_clean_scrub_stamp":"2023-10-30T12:39:53.217887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:20:39.599062+0000","scrub_duration":0.028260459000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":852,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:52.195089+0000","last_change":"2023-10-30T12:39:52.196146+0000","last_active":"2023-10-30T12:39:52.195089+0000","last_peered":"2023-10-30T12:39:52.195089+0000","last_clean":"2023-10-30T12:39:52.195089+0000","last_became_active":"2023-10-30T12:18:01.602190+0000","last_became_peered":"2023-10-30T12:18:01.602190+0000","last_unstale":"2023-10-30T12:39:52.195089+0000","last_undegraded":"2023-10-30T12:39:52.195089+0000","last_fullsized":"2023-10-30T12:39:52.195089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:52.193720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:52.193720+0000","last_clean_scrub_stamp":"2023-10-30T12:39:52.193720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:33:40.108140+0000","scrub_duration":0.043609539000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":858,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:21.782120+0000","last_change":"2023-10-30T12:40:21.782120+0000","last_active":"2023-10-30T12:40:21.782120+0000","last_peered":"2023-10-30T12:40:21.782120+0000","last_clean":"2023-10-30T12:40:21.782120+0000","last_became_active":"2023-10-30T12:19:40.713186+0000","last_became_peered":"2023-10-30T12:19:40.713186+0000","last_unstale":"2023-10-30T12:40:21.782120+0000","last_undegraded":"2023-10-30T12:40:21.782120+0000","last_fullsized":"2023-10-30T12:40:21.782120+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:21.779799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:21.779799+0000","last_clean_scrub_stamp":"2023-10-30T12:40:21.779799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:49:57.747593+0000","scrub_duration":0.060458952000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":863,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:06.663966+0000","last_change":"2023-10-30T12:40:06.665005+0000","last_active":"2023-10-30T12:40:06.663966+0000","last_peered":"2023-10-30T12:40:06.663966+0000","last_clean":"2023-10-30T12:40:06.663966+0000","last_became_active":"2023-10-30T12:18:01.051697+0000","last_became_peered":"2023-10-30T12:18:01.051697+0000","last_unstale":"2023-10-30T12:40:06.663966+0000","last_undegraded":"2023-10-30T12:40:06.663966+0000","last_fullsized":"2023-10-30T12:40:06.663966+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:06.662527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:06.662527+0000","last_clean_scrub_stamp":"2023-10-30T12:40:06.662527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:00:24.353609+0000","scrub_duration":0.032173303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":839,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:57.003604+0000","last_change":"2023-10-30T12:39:57.003604+0000","last_active":"2023-10-30T12:39:57.003604+0000","last_peered":"2023-10-30T12:39:57.003604+0000","last_clean":"2023-10-30T12:39:57.003604+0000","last_became_active":"2023-10-30T12:19:40.864175+0000","last_became_peered":"2023-10-30T12:19:40.864175+0000","last_unstale":"2023-10-30T12:39:57.003604+0000","last_undegraded":"2023-10-30T12:39:57.003604+0000","last_fullsized":"2023-10-30T12:39:57.003604+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:57.001273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:57.001273+0000","last_clean_scrub_stamp":"2023-10-30T12:39:57.001273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:41:07.773438+0000","scrub_duration":0.044252685999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":853,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:56.997999+0000","last_change":"2023-10-30T12:39:56.999005+0000","last_active":"2023-10-30T12:39:56.997999+0000","last_peered":"2023-10-30T12:39:56.997999+0000","last_clean":"2023-10-30T12:39:56.997999+0000","last_became_active":"2023-10-30T12:18:01.523269+0000","last_became_peered":"2023-10-30T12:18:01.523269+0000","last_unstale":"2023-10-30T12:39:56.997999+0000","last_undegraded":"2023-10-30T12:39:56.997999+0000","last_fullsized":"2023-10-30T12:39:56.997999+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:56.996587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:56.996587+0000","last_clean_scrub_stamp":"2023-10-30T12:39:56.996587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:44:13.299852+0000","scrub_duration":0.031392213000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":847,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:03.088506+0000","last_change":"2023-10-30T12:40:03.088506+0000","last_active":"2023-10-30T12:40:03.088506+0000","last_peered":"2023-10-30T12:40:03.088506+0000","last_clean":"2023-10-30T12:40:03.088506+0000","last_became_active":"2023-10-30T12:19:40.440094+0000","last_became_peered":"2023-10-30T12:19:40.440094+0000","last_unstale":"2023-10-30T12:40:03.088506+0000","last_undegraded":"2023-10-30T12:40:03.088506+0000","last_fullsized":"2023-10-30T12:40:03.088506+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:03.085346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:03.085346+0000","last_clean_scrub_stamp":"2023-10-30T12:40:03.085346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:49:12.994355+0000","scrub_duration":0.038565555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":864,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:55.503565+0000","last_change":"2023-10-30T12:39:55.504537+0000","last_active":"2023-10-30T12:39:55.503565+0000","last_peered":"2023-10-30T12:39:55.503565+0000","last_clean":"2023-10-30T12:39:55.503565+0000","last_became_active":"2023-10-30T12:18:01.504543+0000","last_became_peered":"2023-10-30T12:18:01.504543+0000","last_unstale":"2023-10-30T12:39:55.503565+0000","last_undegraded":"2023-10-30T12:39:55.503565+0000","last_fullsized":"2023-10-30T12:39:55.503565+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:55.502243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:55.502243+0000","last_clean_scrub_stamp":"2023-10-30T12:39:55.502243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:59.951192+0000","scrub_duration":0.037490758999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:54.468480+0000","last_change":"2023-10-30T12:39:54.469537+0000","last_active":"2023-10-30T12:39:54.468480+0000","last_peered":"2023-10-30T12:39:54.468480+0000","last_clean":"2023-10-30T12:39:54.468480+0000","last_became_active":"2023-10-30T12:19:29.543275+0000","last_became_peered":"2023-10-30T12:19:29.543275+0000","last_unstale":"2023-10-30T12:39:54.468480+0000","last_undegraded":"2023-10-30T12:39:54.468480+0000","last_fullsized":"2023-10-30T12:39:54.468480+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:54.467054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:54.467054+0000","last_clean_scrub_stamp":"2023-10-30T12:39:54.467054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:07:39.095292+0000","scrub_duration":0.057546617000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":865,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:00.236189+0000","last_change":"2023-10-30T12:40:00.246472+0000","last_active":"2023-10-30T12:40:00.236189+0000","last_peered":"2023-10-30T12:40:00.236189+0000","last_clean":"2023-10-30T12:40:00.236189+0000","last_became_active":"2023-10-30T12:18:00.904525+0000","last_became_peered":"2023-10-30T12:18:00.904525+0000","last_unstale":"2023-10-30T12:40:00.236189+0000","last_undegraded":"2023-10-30T12:40:00.236189+0000","last_fullsized":"2023-10-30T12:40:00.236189+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:00.232185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:00.232185+0000","last_clean_scrub_stamp":"2023-10-30T12:40:00.232185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:35:47.818510+0000","scrub_duration":0.18464655599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":838,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:00.809299+0000","last_change":"2023-10-30T12:40:00.810355+0000","last_active":"2023-10-30T12:40:00.809299+0000","last_peered":"2023-10-30T12:40:00.809299+0000","last_clean":"2023-10-30T12:40:00.809299+0000","last_became_active":"2023-10-30T12:19:26.061689+0000","last_became_peered":"2023-10-30T12:19:26.061689+0000","last_unstale":"2023-10-30T12:40:00.809299+0000","last_undegraded":"2023-10-30T12:40:00.809299+0000","last_fullsized":"2023-10-30T12:40:00.809299+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:00.807878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:00.807878+0000","last_clean_scrub_stamp":"2023-10-30T12:40:00.807878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:54.022399+0000","scrub_duration":0.043296240999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":865,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:53.499813+0000","last_change":"2023-10-30T12:39:53.499813+0000","last_active":"2023-10-30T12:39:53.499813+0000","last_peered":"2023-10-30T12:39:53.499813+0000","last_clean":"2023-10-30T12:39:53.499813+0000","last_became_active":"2023-10-30T12:18:01.003382+0000","last_became_peered":"2023-10-30T12:18:01.003382+0000","last_unstale":"2023-10-30T12:39:53.499813+0000","last_undegraded":"2023-10-30T12:39:53.499813+0000","last_fullsized":"2023-10-30T12:39:53.499813+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:53.493713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:53.493713+0000","last_clean_scrub_stamp":"2023-10-30T12:39:53.493713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:14:45.675107+0000","scrub_duration":0.031685023999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":838,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:53.995696+0000","last_change":"2023-10-30T12:39:53.996734+0000","last_active":"2023-10-30T12:39:53.995696+0000","last_peered":"2023-10-30T12:39:53.995696+0000","last_clean":"2023-10-30T12:39:53.995696+0000","last_became_active":"2023-10-30T12:19:25.272220+0000","last_became_peered":"2023-10-30T12:19:25.272220+0000","last_unstale":"2023-10-30T12:39:53.995696+0000","last_undegraded":"2023-10-30T12:39:53.995696+0000","last_fullsized":"2023-10-30T12:39:53.995696+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:53.994259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:53.994259+0000","last_clean_scrub_stamp":"2023-10-30T12:39:53.994259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:25.256027+0000","scrub_duration":0.091879046000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":864,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:57.570989+0000","last_change":"2023-10-30T12:39:57.573801+0000","last_active":"2023-10-30T12:39:57.570989+0000","last_peered":"2023-10-30T12:39:57.570989+0000","last_clean":"2023-10-30T12:39:57.570989+0000","last_became_active":"2023-10-30T12:18:01.393697+0000","last_became_peered":"2023-10-30T12:18:01.393697+0000","last_unstale":"2023-10-30T12:39:57.570989+0000","last_undegraded":"2023-10-30T12:39:57.570989+0000","last_fullsized":"2023-10-30T12:39:57.570989+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:57.566568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:57.566568+0000","last_clean_scrub_stamp":"2023-10-30T12:39:57.566568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:41:06.942534+0000","scrub_duration":0.048101698999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"900'78","reported_seq":955,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:52.977088+0000","last_change":"2023-10-30T12:39:52.977088+0000","last_active":"2023-10-30T12:39:52.977088+0000","last_peered":"2023-10-30T12:39:52.977088+0000","last_clean":"2023-10-30T12:39:52.977088+0000","last_became_active":"2023-10-30T12:17:56.575838+0000","last_became_peered":"2023-10-30T12:17:56.575838+0000","last_unstale":"2023-10-30T12:39:52.977088+0000","last_undegraded":"2023-10-30T12:39:52.977088+0000","last_fullsized":"2023-10-30T12:39:52.977088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'78","last_scrub_stamp":"2023-10-30T12:39:52.973839+0000","last_deep_scrub":"900'78","last_deep_scrub_stamp":"2023-10-30T12:39:52.973839+0000","last_clean_scrub_stamp":"2023-10-30T12:39:52.973839+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:33:03.900796+0000","scrub_duration":0.218512285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":859,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:19.817265+0000","last_change":"2023-10-30T12:40:19.818269+0000","last_active":"2023-10-30T12:40:19.817265+0000","last_peered":"2023-10-30T12:40:19.817265+0000","last_clean":"2023-10-30T12:40:19.817265+0000","last_became_active":"2023-10-30T12:19:25.715885+0000","last_became_peered":"2023-10-30T12:19:25.715885+0000","last_unstale":"2023-10-30T12:40:19.817265+0000","last_undegraded":"2023-10-30T12:40:19.817265+0000","last_fullsized":"2023-10-30T12:40:19.817265+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:19.815913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:19.815913+0000","last_clean_scrub_stamp":"2023-10-30T12:40:19.815913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:06:33.895576+0000","scrub_duration":0.064070292000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":852,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:02.539109+0000","last_change":"2023-10-30T12:40:02.540166+0000","last_active":"2023-10-30T12:40:02.539109+0000","last_peered":"2023-10-30T12:40:02.539109+0000","last_clean":"2023-10-30T12:40:02.539109+0000","last_became_active":"2023-10-30T12:19:29.879226+0000","last_became_peered":"2023-10-30T12:19:29.879226+0000","last_unstale":"2023-10-30T12:40:02.539109+0000","last_undegraded":"2023-10-30T12:40:02.539109+0000","last_fullsized":"2023-10-30T12:40:02.539109+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:02.537572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:02.537572+0000","last_clean_scrub_stamp":"2023-10-30T12:40:02.537572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:52.257173+0000","scrub_duration":0.046849797999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":856,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:22.792685+0000","last_change":"2023-10-30T12:40:22.793736+0000","last_active":"2023-10-30T12:40:22.792685+0000","last_peered":"2023-10-30T12:40:22.792685+0000","last_clean":"2023-10-30T12:40:22.792685+0000","last_became_active":"2023-10-30T12:19:40.794341+0000","last_became_peered":"2023-10-30T12:19:40.794341+0000","last_unstale":"2023-10-30T12:40:22.792685+0000","last_undegraded":"2023-10-30T12:40:22.792685+0000","last_fullsized":"2023-10-30T12:40:22.792685+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:22.791328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:22.791328+0000","last_clean_scrub_stamp":"2023-10-30T12:40:22.791328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:57.754768+0000","scrub_duration":0.056162954000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":857,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:12.736972+0000","last_change":"2023-10-30T12:40:12.737951+0000","last_active":"2023-10-30T12:40:12.736972+0000","last_peered":"2023-10-30T12:40:12.736972+0000","last_clean":"2023-10-30T12:40:12.736972+0000","last_became_active":"2023-10-30T12:19:28.238513+0000","last_became_peered":"2023-10-30T12:19:28.238513+0000","last_unstale":"2023-10-30T12:40:12.736972+0000","last_undegraded":"2023-10-30T12:40:12.736972+0000","last_fullsized":"2023-10-30T12:40:12.736972+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:12.735713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:12.735713+0000","last_clean_scrub_stamp":"2023-10-30T12:40:12.735713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:04.170559+0000","scrub_duration":0.034735114999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:55.219738+0000","last_change":"2023-10-30T12:39:55.220724+0000","last_active":"2023-10-30T12:39:55.219738+0000","last_peered":"2023-10-30T12:39:55.219738+0000","last_clean":"2023-10-30T12:39:55.219738+0000","last_became_active":"2023-10-30T12:19:27.284772+0000","last_became_peered":"2023-10-30T12:19:27.284772+0000","last_unstale":"2023-10-30T12:39:55.219738+0000","last_undegraded":"2023-10-30T12:39:55.219738+0000","last_fullsized":"2023-10-30T12:39:55.219738+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:55.218406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:55.218406+0000","last_clean_scrub_stamp":"2023-10-30T12:39:55.218406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:00.385740+0000","scrub_duration":0.055291405000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":838,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:59.840386+0000","last_change":"2023-10-30T12:39:59.841454+0000","last_active":"2023-10-30T12:39:59.840386+0000","last_peered":"2023-10-30T12:39:59.840386+0000","last_clean":"2023-10-30T12:39:59.840386+0000","last_became_active":"2023-10-30T12:19:33.483122+0000","last_became_peered":"2023-10-30T12:19:33.483122+0000","last_unstale":"2023-10-30T12:39:59.840386+0000","last_undegraded":"2023-10-30T12:39:59.840386+0000","last_fullsized":"2023-10-30T12:39:59.840386+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:59.837322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:59.837322+0000","last_clean_scrub_stamp":"2023-10-30T12:39:59.837322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:39.954540+0000","scrub_duration":0.041544445999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":853,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:56.509862+0000","last_change":"2023-10-30T12:39:56.510854+0000","last_active":"2023-10-30T12:39:56.509862+0000","last_peered":"2023-10-30T12:39:56.509862+0000","last_clean":"2023-10-30T12:39:56.509862+0000","last_became_active":"2023-10-30T12:19:26.663005+0000","last_became_peered":"2023-10-30T12:19:26.663005+0000","last_unstale":"2023-10-30T12:39:56.509862+0000","last_undegraded":"2023-10-30T12:39:56.509862+0000","last_fullsized":"2023-10-30T12:39:56.509862+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:56.508524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:56.508524+0000","last_clean_scrub_stamp":"2023-10-30T12:39:56.508524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:48.208342+0000","scrub_duration":0.053320273000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":857,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:07.676399+0000","last_change":"2023-10-30T12:40:07.677510+0000","last_active":"2023-10-30T12:40:07.676399+0000","last_peered":"2023-10-30T12:40:07.676399+0000","last_clean":"2023-10-30T12:40:07.676399+0000","last_became_active":"2023-10-30T12:19:26.766234+0000","last_became_peered":"2023-10-30T12:19:26.766234+0000","last_unstale":"2023-10-30T12:40:07.676399+0000","last_undegraded":"2023-10-30T12:40:07.676399+0000","last_fullsized":"2023-10-30T12:40:07.676399+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:07.674810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:07.674810+0000","last_clean_scrub_stamp":"2023-10-30T12:40:07.674810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:45.104913+0000","scrub_duration":0.034958095000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":846,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:56.886448+0000","last_change":"2023-10-30T12:39:56.890537+0000","last_active":"2023-10-30T12:39:56.886448+0000","last_peered":"2023-10-30T12:39:56.886448+0000","last_clean":"2023-10-30T12:39:56.886448+0000","last_became_active":"2023-10-30T12:19:29.191499+0000","last_became_peered":"2023-10-30T12:19:29.191499+0000","last_unstale":"2023-10-30T12:39:56.886448+0000","last_undegraded":"2023-10-30T12:39:56.886448+0000","last_fullsized":"2023-10-30T12:39:56.886448+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:56.885085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:56.885085+0000","last_clean_scrub_stamp":"2023-10-30T12:39:56.885085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:14.236063+0000","scrub_duration":0.059900850999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":859,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:12.154811+0000","last_change":"2023-10-30T12:40:12.155814+0000","last_active":"2023-10-30T12:40:12.154811+0000","last_peered":"2023-10-30T12:40:12.154811+0000","last_clean":"2023-10-30T12:40:12.154811+0000","last_became_active":"2023-10-30T12:19:30.408141+0000","last_became_peered":"2023-10-30T12:19:30.408141+0000","last_unstale":"2023-10-30T12:40:12.154811+0000","last_undegraded":"2023-10-30T12:40:12.154811+0000","last_fullsized":"2023-10-30T12:40:12.154811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:12.153519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:12.153519+0000","last_clean_scrub_stamp":"2023-10-30T12:40:12.153519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:09.258954+0000","scrub_duration":0.045757989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":847,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:40:06.108328+0000","last_change":"2023-10-30T12:40:06.108328+0000","last_active":"2023-10-30T12:40:06.108328+0000","last_peered":"2023-10-30T12:40:06.108328+0000","last_clean":"2023-10-30T12:40:06.108328+0000","last_became_active":"2023-10-30T12:19:40.528642+0000","last_became_peered":"2023-10-30T12:19:40.528642+0000","last_unstale":"2023-10-30T12:40:06.108328+0000","last_undegraded":"2023-10-30T12:40:06.108328+0000","last_fullsized":"2023-10-30T12:40:06.108328+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:40:06.105884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:40:06.105884+0000","last_clean_scrub_stamp":"2023-10-30T12:40:06.105884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:37.288711+0000","scrub_duration":0.055356317000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":848,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:39:59.039684+0000","last_change":"2023-10-30T12:39:59.039684+0000","last_active":"2023-10-30T12:39:59.039684+0000","last_peered":"2023-10-30T12:39:59.039684+0000","last_clean":"2023-10-30T12:39:59.039684+0000","last_became_active":"2023-10-30T12:19:30.720900+0000","last_became_peered":"2023-10-30T12:19:30.720900+0000","last_unstale":"2023-10-30T12:39:59.039684+0000","last_undegraded":"2023-10-30T12:39:59.039684+0000","last_fullsized":"2023-10-30T12:39:59.039684+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:39:59.037328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:39:59.037328+0000","last_clean_scrub_stamp":"2023-10-30T12:39:59.037328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:06:15.048285+0000","scrub_duration":0.04127786,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509715,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62144,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309696,"statfs":{"total":96636764160,"available":96573128704,"internally_reserved":0,"allocated":8970240,"data_stored":4460959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":23.231999999999999,"5min":17.655999999999999,"15min":16.648},"min":{"1min":6.4530000000000003,"5min":4.0250000000000004,"15min":4.0250000000000004},"max":{"1min":181.905,"5min":233.52500000000001,"15min":233.52500000000001},"last":5.9720000000000004},{"interface":"front","average":{"1min":27.143000000000001,"5min":21.59,"15min":22.366},"min":{"1min":9.7650000000000006,"5min":5.6740000000000004,"15min":5.6740000000000004},"max":{"1min":183.072,"5min":214.31899999999999,"15min":365.47399999999999},"last":9.8710000000000004}]},{"osd":1,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.035,"5min":12.183,"15min":14.696},"min":{"1min":6.0800000000000001,"5min":4.4359999999999999,"15min":4.2389999999999999},"max":{"1min":63.286999999999999,"5min":63.286999999999999,"15min":142.51400000000001},"last":12.052},{"interface":"front","average":{"1min":17.978999999999999,"5min":14.823,"15min":17.242000000000001},"min":{"1min":6.056,"5min":5.1689999999999996,"15min":5.1669999999999998},"max":{"1min":61.552,"5min":112.276,"15min":211.11600000000001},"last":8.9779999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":17.715,"5min":21.622,"15min":20.561},"min":{"1min":6.5259999999999998,"5min":6.2530000000000001,"15min":6.2530000000000001},"max":{"1min":61.927,"5min":187.00399999999999,"15min":187.00399999999999},"last":14.474},{"interface":"front","average":{"1min":20.795999999999999,"5min":23.681999999999999,"15min":21.565000000000001},"min":{"1min":8.2710000000000008,"5min":6.5359999999999996,"15min":5.6260000000000003},"max":{"1min":80.686999999999998,"5min":240.99799999999999,"15min":240.99799999999999},"last":10.821}]},{"osd":3,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":20.48,"5min":26.420000000000002,"15min":25.207999999999998},"min":{"1min":10.49,"5min":7.5990000000000002,"15min":6.2240000000000002},"max":{"1min":62.517000000000003,"5min":279.94099999999997,"15min":279.94099999999997},"last":13.022},{"interface":"front","average":{"1min":17.039000000000001,"5min":24.077999999999999,"15min":23.585000000000001},"min":{"1min":8.1370000000000005,"5min":6.617,"15min":6.5960000000000001},"max":{"1min":55.351999999999997,"5min":180.74600000000001,"15min":180.74600000000001},"last":17.399999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":13.237,"5min":19.042000000000002,"15min":18.274999999999999},"min":{"1min":6.617,"5min":6.617,"15min":6.1840000000000002},"max":{"1min":59.103000000000002,"5min":184.88,"15min":184.88},"last":15.298},{"interface":"front","average":{"1min":20.768999999999998,"5min":26.350000000000001,"15min":25.687999999999999},"min":{"1min":11.442,"5min":9.4160000000000004,"15min":7.3540000000000001},"max":{"1min":54.426000000000002,"5min":374.38900000000001,"15min":374.38900000000001},"last":13.707000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":19.882000000000001,"5min":25.530999999999999,"15min":25.718},"min":{"1min":10.507,"5min":8.1210000000000004,"15min":6.2729999999999997},"max":{"1min":43.252000000000002,"5min":218.28800000000001,"15min":218.28800000000001},"last":19.151},{"interface":"front","average":{"1min":17.363,"5min":25.651,"15min":23.794},"min":{"1min":6.7160000000000002,"5min":6.7160000000000002,"15min":6.7160000000000002},"max":{"1min":34.281999999999996,"5min":195.74700000000001,"15min":195.74700000000001},"last":15.983000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:33 2023","interfaces":[{"interface":"back","average":{"1min":21.611999999999998,"5min":23.324000000000002,"15min":27.591999999999999},"min":{"1min":9.8620000000000001,"5min":9.593,"15min":8.3320000000000007},"max":{"1min":69.704999999999998,"5min":178.68100000000001,"15min":224.71000000000001},"last":20.007999999999999},{"interface":"front","average":{"1min":19.506,"5min":24.533000000000001,"15min":25.074000000000002},"min":{"1min":9.7799999999999994,"5min":7.2469999999999999,"15min":7.2469999999999999},"max":{"1min":39.325000000000003,"5min":187.14500000000001,"15min":210.05199999999999},"last":16.632999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80780,"kb_used_data":9348,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94291060,"statfs":{"total":96636764160,"available":96554045440,"internally_reserved":0,"allocated":9572352,"data_stored":4982006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:46 2023","interfaces":[{"interface":"back","average":{"1min":13.422000000000001,"5min":16.344999999999999,"15min":23.686},"min":{"1min":4.9379999999999997,"5min":4.1050000000000004,"15min":4.1050000000000004},"max":{"1min":61.497999999999998,"5min":158.87799999999999,"15min":308.51999999999998},"last":5.242},{"interface":"front","average":{"1min":15.584,"5min":17.266999999999999,"15min":22.806999999999999},"min":{"1min":5.7670000000000003,"5min":4.7469999999999999,"15min":4.7469999999999999},"max":{"1min":66.950999999999993,"5min":146.374,"15min":244.072},"last":7.6130000000000004}]},{"osd":1,"last update":"Mon Oct 30 12:39:56 2023","interfaces":[{"interface":"back","average":{"1min":24.869,"5min":18.227,"15min":24.437000000000001},"min":{"1min":4.9160000000000004,"5min":4.9160000000000004,"15min":4.9160000000000004},"max":{"1min":227.51499999999999,"5min":227.51499999999999,"15min":304.13299999999998},"last":9.4090000000000007},{"interface":"front","average":{"1min":22.501999999999999,"5min":18.318000000000001,"15min":23.754999999999999},"min":{"1min":7.8200000000000003,"5min":5.6150000000000002,"15min":5.4109999999999996},"max":{"1min":102.226,"5min":162.25200000000001,"15min":278.33999999999997},"last":10.662000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:39:56 2023","interfaces":[{"interface":"back","average":{"1min":26.420000000000002,"5min":20.91,"15min":22.443000000000001},"min":{"1min":5.6989999999999998,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":219.33600000000001,"5min":219.33600000000001,"15min":246.37},"last":13.253},{"interface":"front","average":{"1min":31.077999999999999,"5min":24.548999999999999,"15min":23.369},"min":{"1min":9.3379999999999992,"5min":8.4540000000000006,"15min":5.2770000000000001},"max":{"1min":229.303,"5min":229.303,"15min":303.274},"last":12.428000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:40:26 2023","interfaces":[{"interface":"back","average":{"1min":16.629999999999999,"5min":21.940000000000001,"15min":23.689},"min":{"1min":9.8979999999999997,"5min":6.3860000000000001,"15min":6.3860000000000001},"max":{"1min":61.011000000000003,"5min":148.125,"15min":273.10300000000001},"last":16.888999999999999},{"interface":"front","average":{"1min":21.023,"5min":21.888999999999999,"15min":23.698},"min":{"1min":9.3279999999999994,"5min":6.2889999999999997,"15min":5.8029999999999999},"max":{"1min":91.418000000000006,"5min":154.857,"15min":282.28500000000003},"last":13.906000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:39:56 2023","interfaces":[{"interface":"back","average":{"1min":20.756,"5min":22.402999999999999,"15min":27.849},"min":{"1min":11.57,"5min":9.5109999999999992,"15min":7.2830000000000004},"max":{"1min":92.058000000000007,"5min":144.102,"15min":338.60000000000002},"last":14.957000000000001},{"interface":"front","average":{"1min":17.724,"5min":24.602,"15min":27.478999999999999},"min":{"1min":7.4420000000000002,"5min":7.4420000000000002,"15min":6.3890000000000002},"max":{"1min":97.159999999999997,"5min":169.45500000000001,"15min":295.76600000000002},"last":11.313000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:56 2023","interfaces":[{"interface":"back","average":{"1min":21.257000000000001,"5min":25.925999999999998,"15min":28.923999999999999},"min":{"1min":10.871,"5min":10.871,"15min":7.9729999999999999},"max":{"1min":94.408000000000001,"5min":178.637,"15min":255.291},"last":15.554},{"interface":"front","average":{"1min":24.161999999999999,"5min":31.087,"15min":34.582999999999998},"min":{"1min":13.163,"5min":12.661,"15min":7.5430000000000001},"max":{"1min":95.777000000000001,"5min":319.17700000000002,"15min":320.678},"last":16.149999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:40:26 2023","interfaces":[{"interface":"back","average":{"1min":17.254999999999999,"5min":25.902000000000001,"15min":27.509},"min":{"1min":6.6710000000000003,"5min":6.6710000000000003,"15min":6.6710000000000003},"max":{"1min":79.741,"5min":258.61200000000002,"15min":258.61200000000002},"last":9.7360000000000007},{"interface":"front","average":{"1min":17.495000000000001,"5min":21.068000000000001,"15min":23.629000000000001},"min":{"1min":8.6430000000000007,"5min":6.109,"15min":6.109},"max":{"1min":63.814999999999998,"5min":143.20500000000001,"15min":245.71899999999999},"last":18.273}]}]},{"osd":4,"up_from":15,"seq":64424509716,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62144,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309696,"statfs":{"total":96636764160,"available":96573128704,"internally_reserved":0,"allocated":8970240,"data_stored":4460959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":8.8330000000000002,"5min":15.872,"15min":22.242999999999999},"min":{"1min":4.7569999999999997,"5min":4.2140000000000004,"15min":4.0620000000000003},"max":{"1min":16.516999999999999,"5min":158.499,"15min":295.56400000000002},"last":7.5140000000000002},{"interface":"front","average":{"1min":8.3000000000000007,"5min":18.66,"15min":23.565999999999999},"min":{"1min":4.4080000000000004,"5min":4.4080000000000004,"15min":4.4080000000000004},"max":{"1min":14.081,"5min":182.96600000000001,"15min":368.01900000000001},"last":9.8360000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:39:46 2023","interfaces":[{"interface":"back","average":{"1min":11.259,"5min":16.731000000000002,"15min":20.905999999999999},"min":{"1min":6.633,"5min":5.6420000000000003,"15min":4.5389999999999997},"max":{"1min":18.992999999999999,"5min":171.24100000000001,"15min":322.97800000000001},"last":17.324999999999999},{"interface":"front","average":{"1min":15.079000000000001,"5min":21.649000000000001,"15min":20.692},"min":{"1min":6.6539999999999999,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":32.031999999999996,"5min":168.833,"15min":245.44900000000001},"last":8.6560000000000006}]},{"osd":2,"last update":"Mon Oct 30 12:39:54 2023","interfaces":[{"interface":"back","average":{"1min":15.304,"5min":26.896999999999998,"15min":23.456},"min":{"1min":6.8570000000000002,"5min":6.641,"15min":6.0039999999999996},"max":{"1min":60.234999999999999,"5min":316.74799999999999,"15min":316.74799999999999},"last":10.568},{"interface":"front","average":{"1min":17.652999999999999,"5min":26.408000000000001,"15min":25.983000000000001},"min":{"1min":6.7469999999999999,"5min":5.9489999999999998,"15min":5.9489999999999998},"max":{"1min":52.188000000000002,"5min":317.55399999999997,"15min":317.55399999999997},"last":18.364999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":14.321999999999999,"5min":23.728999999999999,"15min":28.404},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":6.7270000000000003},"max":{"1min":21.91,"5min":161.197,"15min":383.87099999999998},"last":11.227},{"interface":"front","average":{"1min":13.757999999999999,"5min":26.026,"15min":26.616},"min":{"1min":8.5,"5min":8.5,"15min":6.6529999999999996},"max":{"1min":18.109999999999999,"5min":260.94200000000001,"15min":284.61700000000002},"last":19.213000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:39:46 2023","interfaces":[{"interface":"back","average":{"1min":19.042999999999999,"5min":27.260999999999999,"15min":27.414000000000001},"min":{"1min":8.2910000000000004,"5min":7.9429999999999996,"15min":5.7119999999999997},"max":{"1min":51.045999999999999,"5min":185.47200000000001,"15min":261.80200000000002},"last":20.123000000000001},{"interface":"front","average":{"1min":16.626999999999999,"5min":21.643000000000001,"15min":26.568000000000001},"min":{"1min":8.0180000000000007,"5min":6.3760000000000003,"15min":6.3760000000000003},"max":{"1min":33.390999999999998,"5min":174.38999999999999,"15min":371.35700000000003},"last":12.112}]},{"osd":6,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":23.914999999999999,"5min":27.789999999999999,"15min":27.952999999999999},"min":{"1min":10.460000000000001,"5min":9.4939999999999998,"15min":8.0060000000000002},"max":{"1min":95.795000000000002,"5min":215.31800000000001,"15min":312.44999999999999},"last":21.210999999999999},{"interface":"front","average":{"1min":19.725999999999999,"5min":28.399999999999999,"15min":27.664000000000001},"min":{"1min":9.859,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":63.609999999999999,"5min":266.50599999999997,"15min":320.50999999999999},"last":14.651999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:39:34 2023","interfaces":[{"interface":"back","average":{"1min":17.331,"5min":26.408000000000001,"15min":28.349},"min":{"1min":9.048,"5min":9.048,"15min":7.9370000000000003},"max":{"1min":55.015000000000001,"5min":193.95400000000001,"15min":319.84199999999998},"last":13.473000000000001},{"interface":"front","average":{"1min":15.836,"5min":26.617000000000001,"15min":27.077000000000002},"min":{"1min":10.515000000000001,"5min":8.9489999999999998,"15min":8.5410000000000004},"max":{"1min":28.527000000000001,"5min":252.232,"15min":315.06900000000002},"last":16.352}]}]},{"osd":3,"up_from":15,"seq":64424509713,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84352,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287488,"statfs":{"total":96636764160,"available":96550387712,"internally_reserved":0,"allocated":8970240,"data_stored":4460959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.731999999999999,"5min":21.190000000000001,"15min":21.754999999999999},"min":{"1min":4.609,"5min":4.2460000000000004,"15min":4.1159999999999997},"max":{"1min":210.398,"5min":303.58600000000001,"15min":303.58600000000001},"last":7.54},{"interface":"front","average":{"1min":26.295999999999999,"5min":17.559999999999999,"15min":21.213000000000001},"min":{"1min":6.0049999999999999,"5min":4.8819999999999997,"15min":4.1070000000000002},"max":{"1min":187.82400000000001,"5min":187.82400000000001,"15min":268.68799999999999},"last":6.8460000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":42.889000000000003,"5min":25.684999999999999,"15min":23.533999999999999},"min":{"1min":7.7990000000000004,"5min":5.6070000000000002,"15min":4.8940000000000001},"max":{"1min":220.96600000000001,"5min":220.96600000000001,"15min":389.303},"last":6.0190000000000001},{"interface":"front","average":{"1min":43.094999999999999,"5min":25.721,"15min":24.445},"min":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":5.0890000000000004},"max":{"1min":222.328,"5min":222.328,"15min":468.13299999999998},"last":10.119}]},{"osd":2,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":19.759,"5min":21.931000000000001,"15min":22.350999999999999},"min":{"1min":8.0250000000000004,"5min":6.8230000000000004,"15min":5.6150000000000002},"max":{"1min":135.97399999999999,"5min":352.09399999999999,"15min":352.09399999999999},"last":8.3249999999999993},{"interface":"front","average":{"1min":18.385000000000002,"5min":20.425999999999998,"15min":23.646999999999998},"min":{"1min":7.7889999999999997,"5min":6.9850000000000003,"15min":6.9850000000000003},"max":{"1min":94.552999999999997,"5min":184.04599999999999,"15min":287.33699999999999},"last":11.138999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":32.673000000000002,"5min":25.640000000000001,"15min":28.405000000000001},"min":{"1min":8.6929999999999996,"5min":7.1799999999999997,"15min":6.5010000000000003},"max":{"1min":211.03200000000001,"5min":310.12099999999998,"15min":310.12099999999998},"last":13.711},{"interface":"front","average":{"1min":31.611000000000001,"5min":25.715,"15min":25.268000000000001},"min":{"1min":9.2690000000000001,"5min":6.7169999999999996,"15min":6.0919999999999996},"max":{"1min":187.15700000000001,"5min":359.87799999999999,"15min":359.87799999999999},"last":12.537000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":44.430999999999997,"5min":29.558,"15min":30.042999999999999},"min":{"1min":9.8840000000000003,"5min":9,"15min":6.851},"max":{"1min":274.15300000000002,"5min":360.88099999999997,"15min":564.26700000000005},"last":13.128},{"interface":"front","average":{"1min":50.723999999999997,"5min":28.849,"15min":28.324999999999999},"min":{"1min":10.619,"5min":8.8680000000000003,"15min":8.0109999999999992},"max":{"1min":289.68099999999998,"5min":289.68099999999998,"15min":294.392},"last":15.646000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":30.876999999999999,"5min":29.062999999999999,"15min":30.353999999999999},"min":{"1min":11.016,"5min":9.3369999999999997,"15min":8.8149999999999995},"max":{"1min":197.09200000000001,"5min":374.33999999999997,"15min":374.33999999999997},"last":14.942},{"interface":"front","average":{"1min":33.301000000000002,"5min":31.440000000000001,"15min":31.552},"min":{"1min":11.416,"5min":11.416,"15min":8.9079999999999995},"max":{"1min":177.63499999999999,"5min":352.79500000000002,"15min":352.79500000000002},"last":16.724}]},{"osd":7,"last update":"Mon Oct 30 12:39:27 2023","interfaces":[{"interface":"back","average":{"1min":36.420000000000002,"5min":32.140999999999998,"15min":34.933},"min":{"1min":12.868,"5min":11.401999999999999,"15min":8.5099999999999998},"max":{"1min":203.70099999999999,"5min":382.68299999999999,"15min":382.68299999999999},"last":18.048999999999999},{"interface":"front","average":{"1min":24.998999999999999,"5min":29.645,"15min":32.134},"min":{"1min":11.976000000000001,"5min":9.0850000000000009,"15min":8.7699999999999996},"max":{"1min":86.024000000000001,"5min":361.47199999999998,"15min":562.05700000000002},"last":17.408999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509714,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65920,"kb_used_data":8376,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94305920,"statfs":{"total":96636764160,"available":96569262080,"internally_reserved":0,"allocated":8577024,"data_stored":4196328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:40:02 2023","interfaces":[{"interface":"back","average":{"1min":16.411999999999999,"5min":15.106999999999999,"15min":18.882000000000001},"min":{"1min":4.1559999999999997,"5min":4.1559999999999997,"15min":3.8079999999999998},"max":{"1min":72.049000000000007,"5min":119.291,"15min":337.86700000000002},"last":5.7190000000000003},{"interface":"front","average":{"1min":22.454000000000001,"5min":15.361000000000001,"15min":17.914000000000001},"min":{"1min":6.2690000000000001,"5min":4.4279999999999999,"15min":4.1399999999999997},"max":{"1min":137.864,"5min":137.864,"15min":343.983},"last":6.4809999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:39:54 2023","interfaces":[{"interface":"back","average":{"1min":27.044,"5min":18.896999999999998,"15min":19.431999999999999},"min":{"1min":8.4809999999999999,"5min":4.9790000000000001,"15min":4.6559999999999997},"max":{"1min":221.81200000000001,"5min":221.81200000000001,"15min":312.11900000000003},"last":10.651},{"interface":"front","average":{"1min":24.356000000000002,"5min":19.363,"15min":18.806999999999999},"min":{"1min":6.9180000000000001,"5min":6.9180000000000001,"15min":6.649},"max":{"1min":214.18100000000001,"5min":214.18100000000001,"15min":311.22800000000001},"last":17.824000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:39:59 2023","interfaces":[{"interface":"back","average":{"1min":19.18,"5min":22.670000000000002,"15min":23.629000000000001},"min":{"1min":7,"5min":7,"15min":6.6379999999999999},"max":{"1min":73.805000000000007,"5min":173.56899999999999,"15min":337.33100000000002},"last":17.007999999999999},{"interface":"front","average":{"1min":30.245999999999999,"5min":24.972000000000001,"15min":24.486999999999998},"min":{"1min":9.2439999999999998,"5min":7.3620000000000001,"15min":6.6269999999999998},"max":{"1min":196.02099999999999,"5min":197.904,"15min":314.721},"last":12.141}]},{"osd":4,"last update":"Mon Oct 30 12:39:42 2023","interfaces":[{"interface":"back","average":{"1min":16.821999999999999,"5min":22.812000000000001,"15min":23.850999999999999},"min":{"1min":8,"5min":8,"15min":6.415},"max":{"1min":40.204999999999998,"5min":157.77199999999999,"15min":331.834},"last":18.771000000000001},{"interface":"front","average":{"1min":15.441000000000001,"5min":17.044,"15min":18.234999999999999},"min":{"1min":6.8239999999999998,"5min":6.1509999999999998,"15min":5.984},"max":{"1min":47.32,"5min":134.452,"15min":179.89099999999999},"last":9.0380000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:39:47 2023","interfaces":[{"interface":"back","average":{"1min":19.218,"5min":22.625,"15min":28.466999999999999},"min":{"1min":9.5459999999999994,"5min":8.0619999999999994,"15min":7.4500000000000002},"max":{"1min":42.088000000000001,"5min":201.00299999999999,"15min":326.47699999999998},"last":13.59},{"interface":"front","average":{"1min":18.844999999999999,"5min":24.981000000000002,"15min":30.864999999999998},"min":{"1min":8.9949999999999992,"5min":8.9949999999999992,"15min":8.9949999999999992},"max":{"1min":40.993000000000002,"5min":238.57300000000001,"15min":338.82100000000003},"last":19.399999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:40:30 2023","interfaces":[{"interface":"back","average":{"1min":13.832000000000001,"5min":17.437999999999999,"15min":18.591999999999999},"min":{"1min":8.4309999999999992,"5min":6.3239999999999998,"15min":6.1100000000000003},"max":{"1min":33.173000000000002,"5min":146.75,"15min":168.20400000000001},"last":9.6649999999999991},{"interface":"front","average":{"1min":18.831,"5min":21.277999999999999,"15min":24.571999999999999},"min":{"1min":14.038,"5min":9.8819999999999997,"15min":8.7729999999999997},"max":{"1min":37.017000000000003,"5min":109.34699999999999,"15min":315.71100000000001},"last":14.564}]},{"osd":7,"last update":"Mon Oct 30 12:39:54 2023","interfaces":[{"interface":"back","average":{"1min":19.215,"5min":19.186,"15min":19.881},"min":{"1min":8.2750000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"max":{"1min":47.933,"5min":123.926,"15min":228.67400000000001},"last":11.462999999999999},{"interface":"front","average":{"1min":23.645,"5min":25.620000000000001,"15min":26.815000000000001},"min":{"1min":15.757999999999999,"5min":10.462999999999999,"15min":9.0589999999999993},"max":{"1min":39.351999999999997,"5min":160.93600000000001,"15min":328.51799999999997},"last":15.541}]}]},{"osd":1,"up_from":15,"seq":64424509714,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62728,"kb_used_data":9344,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309112,"statfs":{"total":96636764160,"available":96572530688,"internally_reserved":0,"allocated":9568256,"data_stored":4982001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":10.207000000000001,"5min":11.294,"15min":18.631},"min":{"1min":5.5519999999999996,"5min":4.3760000000000003,"15min":4.3760000000000003},"max":{"1min":24.184999999999999,"5min":75.006,"15min":294.67099999999999},"last":7.173},{"interface":"front","average":{"1min":11.981999999999999,"5min":11.672000000000001,"15min":17.254000000000001},"min":{"1min":6.4210000000000003,"5min":5.2400000000000002,"15min":4.952},"max":{"1min":27.530999999999999,"5min":61.932000000000002,"15min":241.066},"last":7.8470000000000004}]},{"osd":2,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":16.41,"5min":14.955,"15min":19.184000000000001},"min":{"1min":6.4299999999999997,"5min":6.4299999999999997,"15min":6.0519999999999996},"max":{"1min":67.140000000000001,"5min":75.491,"15min":295.50599999999997},"last":9.6620000000000008},{"interface":"front","average":{"1min":18.949999999999999,"5min":13.51,"15min":19.122},"min":{"1min":6.8810000000000002,"5min":6.1580000000000004,"15min":5.327},"max":{"1min":135.00800000000001,"5min":135.00800000000001,"15min":269.99200000000002},"last":6.359}]},{"osd":3,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":19.390000000000001,"15min":21.367999999999999},"min":{"1min":6.9889999999999999,"5min":6.4779999999999998,"15min":6.4779999999999998},"max":{"1min":25.774999999999999,"5min":310.40800000000002,"15min":551.58000000000004},"last":8.9499999999999993},{"interface":"front","average":{"1min":17.298999999999999,"5min":28.062000000000001,"15min":27.957000000000001},"min":{"1min":9.8819999999999997,"5min":9.8819999999999997,"15min":9.1999999999999993},"max":{"1min":29.579000000000001,"5min":338.88600000000002,"15min":562.06899999999996},"last":12.099}]},{"osd":4,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":14.186,"5min":16.617000000000001,"15min":21.454000000000001},"min":{"1min":7.0469999999999997,"5min":6.7480000000000002,"15min":6.6109999999999998},"max":{"1min":40.465000000000003,"5min":84.212000000000003,"15min":285.26499999999999},"last":13.67},{"interface":"front","average":{"1min":14.521000000000001,"5min":16.108000000000001,"15min":22.126000000000001},"min":{"1min":9.0500000000000007,"5min":6.7450000000000001,"15min":6.7450000000000001},"max":{"1min":42.087000000000003,"5min":62.616,"15min":327.59699999999998},"last":11.512}]},{"osd":5,"last update":"Mon Oct 30 12:39:38 2023","interfaces":[{"interface":"back","average":{"1min":35.115000000000002,"5min":25.553999999999998,"15min":26.443999999999999},"min":{"1min":9.9410000000000007,"5min":8.1639999999999997,"15min":7.3070000000000004},"max":{"1min":322.59899999999999,"5min":322.59899999999999,"15min":322.59899999999999},"last":10.869999999999999},{"interface":"front","average":{"1min":32.095999999999997,"5min":21.616,"15min":24.370999999999999},"min":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"max":{"1min":283.06,"5min":283.06,"15min":286.43799999999999},"last":10.278}]},{"osd":6,"last update":"Mon Oct 30 12:39:29 2023","interfaces":[{"interface":"back","average":{"1min":13.723000000000001,"5min":17.039000000000001,"15min":22.716000000000001},"min":{"1min":7.6059999999999999,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":28.151,"5min":71.662999999999997,"15min":348.91000000000003},"last":15.015000000000001},{"interface":"front","average":{"1min":18.815999999999999,"5min":23.582999999999998,"15min":26.449000000000002},"min":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":8.4819999999999993},"max":{"1min":33.466000000000001,"5min":99.231999999999999,"15min":301.99799999999999},"last":15.679}]},{"osd":7,"last update":"Mon Oct 30 12:39:37 2023","interfaces":[{"interface":"back","average":{"1min":21.850999999999999,"5min":21.949000000000002,"15min":25.800999999999998},"min":{"1min":13.108000000000001,"5min":9.2780000000000005,"15min":7.8739999999999997},"max":{"1min":68.974999999999994,"5min":107.703,"15min":305.19900000000001},"last":14.315},{"interface":"front","average":{"1min":20.207000000000001,"5min":20.809000000000001,"15min":26.821000000000002},"min":{"1min":8.8640000000000008,"5min":8.8640000000000008,"15min":8.0920000000000005},"max":{"1min":70.406999999999996,"5min":106.678,"15min":277.05200000000002},"last":16.986000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509714,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62148,"kb_used_data":8764,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309692,"statfs":{"total":96636764160,"available":96573124608,"internally_reserved":0,"allocated":8974336,"data_stored":4460964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":14.013,"5min":16.620999999999999,"15min":19.768000000000001},"min":{"1min":6.117,"5min":4.1799999999999997,"15min":4.1799999999999997},"max":{"1min":73.113,"5min":232.798,"15min":287.096},"last":8.1310000000000002},{"interface":"front","average":{"1min":11.015000000000001,"5min":14.513,"15min":22.515999999999998},"min":{"1min":7.6879999999999997,"5min":5.2370000000000001,"15min":4.7590000000000003},"max":{"1min":29.097000000000001,"5min":218.61099999999999,"15min":425.762},"last":10.215}]},{"osd":2,"last update":"Mon Oct 30 12:39:40 2023","interfaces":[{"interface":"back","average":{"1min":21.314,"5min":17.907,"15min":20.555},"min":{"1min":8.3599999999999994,"5min":5.2480000000000002,"15min":5.2480000000000002},"max":{"1min":78.947999999999993,"5min":228.90799999999999,"15min":299.46600000000001},"last":11.342000000000001},{"interface":"front","average":{"1min":28.640000000000001,"5min":18.164000000000001,"15min":20.786000000000001},"min":{"1min":6.7779999999999996,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":113.76000000000001,"5min":138.13499999999999,"15min":258.45400000000001},"last":10.612}]},{"osd":3,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.376999999999999,"5min":16.640999999999998,"15min":20.597999999999999},"min":{"1min":9.6329999999999991,"5min":6.5499999999999998,"15min":5.2110000000000003},"max":{"1min":41.456000000000003,"5min":166.21600000000001,"15min":264.97199999999998},"last":12.686},{"interface":"front","average":{"1min":19.736000000000001,"5min":21.084,"15min":25.367000000000001},"min":{"1min":8.4079999999999995,"5min":8.4079999999999995,"15min":7.218},"max":{"1min":54.859000000000002,"5min":230.73699999999999,"15min":258.93200000000002},"last":11.978}]},{"osd":4,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6620000000000008,"5min":18.376000000000001,"15min":22.655999999999999},"min":{"1min":5.681,"5min":5.681,"15min":5.681},"max":{"1min":24.405999999999999,"5min":209.64599999999999,"15min":209.64599999999999},"last":7.6379999999999999},{"interface":"front","average":{"1min":20.053999999999998,"5min":25.309000000000001,"15min":23.466999999999999},"min":{"1min":8.8819999999999997,"5min":8.0990000000000002,"15min":5.8499999999999996},"max":{"1min":90.787999999999997,"5min":208.92400000000001,"15min":208.92400000000001},"last":16.260000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":17.98,"5min":29.106000000000002,"15min":29.109999999999999},"min":{"1min":10.965,"5min":8.9440000000000008,"15min":7.5510000000000002},"max":{"1min":31.939,"5min":245.16300000000001,"15min":285.57100000000003},"last":16.91},{"interface":"front","average":{"1min":17.673999999999999,"5min":21.956,"15min":28.129999999999999},"min":{"1min":10.518000000000001,"5min":7.2720000000000002,"15min":7.2720000000000002},"max":{"1min":30.361000000000001,"5min":142.25,"15min":292.07900000000001},"last":13.317}]},{"osd":6,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":15.827999999999999,"5min":20.135999999999999,"15min":21.259},"min":{"1min":7.7000000000000002,"5min":7.4370000000000003,"15min":7.4370000000000003},"max":{"1min":43.283999999999999,"5min":171.99600000000001,"15min":236.40000000000001},"last":8.702},{"interface":"front","average":{"1min":18.968,"5min":24.116,"15min":25.454999999999998},"min":{"1min":12.252000000000001,"5min":10.470000000000001,"15min":8.9199999999999999},"max":{"1min":31.385000000000002,"5min":178.59999999999999,"15min":220.68700000000001},"last":15.352}]},{"osd":7,"last update":"Mon Oct 30 12:39:31 2023","interfaces":[{"interface":"back","average":{"1min":19.300999999999998,"5min":20.318000000000001,"15min":24.036000000000001},"min":{"1min":11.853,"5min":9.2100000000000009,"15min":9.2100000000000009},"max":{"1min":32.664999999999999,"5min":143.49600000000001,"15min":245.87100000000001},"last":13.927},{"interface":"front","average":{"1min":16.231000000000002,"5min":22.702000000000002,"15min":24.879999999999999},"min":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":32.033000000000001,"5min":181.15100000000001,"15min":260.04000000000002},"last":17.928999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575122,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61760,"kb_used_data":8376,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310080,"statfs":{"total":96636764160,"available":96573521920,"internally_reserved":0,"allocated":8577024,"data_stored":4196328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":14.144,"5min":11.411,"15min":20.879000000000001},"min":{"1min":4.5,"5min":4.0960000000000001,"15min":4.0099999999999998},"max":{"1min":110.503,"5min":110.503,"15min":623.43700000000001},"last":10.103},{"interface":"front","average":{"1min":10.115,"5min":11.086,"15min":18.956},"min":{"1min":4.9050000000000002,"5min":4.202,"15min":4.202},"max":{"1min":36.860999999999997,"5min":51.253999999999998,"15min":563.08900000000006},"last":7.3680000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":16.497,"5min":13.163,"15min":16.117000000000001},"min":{"1min":6.3159999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"max":{"1min":143.386,"5min":143.386,"15min":159.941},"last":8.8019999999999996},{"interface":"front","average":{"1min":21.492000000000001,"5min":17.748000000000001,"15min":21.466000000000001},"min":{"1min":9.5489999999999995,"5min":7.0679999999999996,"15min":7.008},"max":{"1min":153.08199999999999,"5min":153.08199999999999,"15min":266.30599999999998},"last":17.398}]},{"osd":2,"last update":"Mon Oct 30 12:39:51 2023","interfaces":[{"interface":"back","average":{"1min":22.289999999999999,"5min":20.649000000000001,"15min":20.420999999999999},"min":{"1min":6.5949999999999998,"5min":6.1749999999999998,"15min":5.0529999999999999},"max":{"1min":190.024,"5min":329.81999999999999,"15min":329.81999999999999},"last":16.5},{"interface":"front","average":{"1min":14.016,"5min":16.754000000000001,"15min":20.099},"min":{"1min":7.9550000000000001,"5min":6.8129999999999997,"15min":6.0350000000000001},"max":{"1min":51.386000000000003,"5min":132.25800000000001,"15min":132.25800000000001},"last":10.776}]},{"osd":3,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":20.202999999999999,"5min":19.655999999999999,"15min":23.643000000000001},"min":{"1min":8.0839999999999996,"5min":6.4950000000000001,"15min":6.4950000000000001},"max":{"1min":124.31399999999999,"5min":124.31399999999999,"15min":185.98400000000001},"last":11.493},{"interface":"front","average":{"1min":22.238,"5min":22.606999999999999,"15min":24.629999999999999},"min":{"1min":8.4100000000000001,"5min":7.7000000000000002,"15min":6.8159999999999998},"max":{"1min":159.69999999999999,"5min":247.65899999999999,"15min":247.65899999999999},"last":15.635}]},{"osd":4,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":17.640000000000001,"5min":19.411000000000001,"15min":24.236999999999998},"min":{"1min":9.4979999999999993,"5min":5.8150000000000004,"15min":5.8150000000000004},"max":{"1min":50.786000000000001,"5min":117.967,"15min":232.17099999999999},"last":12.824},{"interface":"front","average":{"1min":18.247,"5min":18.09,"15min":22.715},"min":{"1min":6.806,"5min":6.806,"15min":6.6280000000000001},"max":{"1min":100.952,"5min":100.952,"15min":267.11000000000001},"last":9.4649999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":20.390000000000001,"5min":22.111000000000001,"15min":27.120999999999999},"min":{"1min":7.6349999999999998,"5min":7.1440000000000001,"15min":6.5110000000000001},"max":{"1min":125.926,"5min":186.54499999999999,"15min":203.66399999999999},"last":12.173999999999999},{"interface":"front","average":{"1min":16.417999999999999,"5min":23.021999999999998,"15min":27.204000000000001},"min":{"1min":8.6980000000000004,"5min":8.6980000000000004,"15min":7.1500000000000004},"max":{"1min":46.524000000000001,"5min":185.61099999999999,"15min":275.88099999999997},"last":19.010999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:39:41 2023","interfaces":[{"interface":"back","average":{"1min":18.094000000000001,"5min":20.02,"15min":26.045000000000002},"min":{"1min":8.952,"5min":8.952,"15min":7.5579999999999998},"max":{"1min":50.177999999999997,"5min":99.480999999999995,"15min":227.18600000000001},"last":13.452999999999999},{"interface":"front","average":{"1min":17.981000000000002,"5min":21.117000000000001,"15min":25.186},"min":{"1min":9.5609999999999999,"5min":8.8019999999999996,"15min":8.8019999999999996},"max":{"1min":51.954000000000001,"5min":108.705,"15min":264.99299999999999},"last":18.201000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:40:36.189 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T12:40:36.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.288+0000 7f061b8d0640 10 monclient: get_monmap_and_config 2023-10-30T12:40:36.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.288+0000 7f061b8d0640 10 monclient: build_initial_monmap 2023-10-30T12:40:36.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.288+0000 7f061b8d0640 10 monclient: monmap: 2023-10-30T12:40:36.290 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:40:36.290 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:36.290 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:40:36.289151+0000 2023-10-30T12:40:36.291 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:40:36.289151+0000 2023-10-30T12:40:36.291 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:40:36.291 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:36.291 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a-legacy 2023-10-30T12:40:36.291 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2023-10-30T12:40:36.292 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-b 2023-10-30T12:40:36.292 INFO:teuthology.orchestra.run.smithi148.stderr:3: v2:172.21.15.148:3300/0 mon.noname-a 2023-10-30T12:40:36.292 INFO:teuthology.orchestra.run.smithi148.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2023-10-30T12:40:36.292 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:36.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.288+0000 7f061b8d0640 1 Processor -- start 2023-10-30T12:40:36.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.288+0000 7f061b8d0640 1 -- start start 2023-10-30T12:40:36.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.288+0000 7f061b8d0640 10 monclient: init 2023-10-30T12:40:36.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:40:36.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-10-30T12:40:36.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06140b7c10 addr v1:172.21.15.150:6789/0 2023-10-30T12:40:36.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0614141940 0x7f0614141d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 10 monclient(hunting): picked mon.noname-a con 0x7f0614141940 addr v2:172.21.15.148:3300/0 2023-10-30T12:40:36.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f061413e730 addr v1:172.21.15.148:6789/0 2023-10-30T12:40:36.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06140a9350 con 0x7f061413e730 2023-10-30T12:40:36.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06140aa6a0 con 0x7f06140b7c10 2023-10-30T12:40:36.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:36.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06140670a0 con 0x7f0614141940 2023-10-30T12:40:36.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:40:36.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.289+0000 7f061b8d0640 10 monclient(hunting): authenticate will time out at 2928.494430s 2023-10-30T12:40:36.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.290+0000 7f061966c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f06140b7c10 0x7f06140a8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:37544/0 (socket says 172.21.15.148:37544) 2023-10-30T12:40:36.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.290+0000 7f061966c640 1 -- 172.21.15.148:0/2913076767 learned_addr learned my addr 172.21.15.148:0/2913076767 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T12:40:36.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.290+0000 7f0618e6b640 1 --2- 172.21.15.148:0/2913076767 >> v2:172.21.15.148:3300/0 conn(0x7f0614141940 0x7f0614141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.291+0000 7f0618e6b640 10 monclient(hunting): get_auth_request con 0x7f0614141940 auth_method 0 2023-10-30T12:40:36.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.291+0000 7f0618e6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:36.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.291+0000 7f0618e6b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:36.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.291+0000 7f0618e6b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:36.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.292+0000 7f0613fff640 1 -- 172.21.15.148:0/2913076767 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238105997 0 0) 0x7f0604001490 con 0x7f06140b7c10 2023-10-30T12:40:36.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.292+0000 7f0613fff640 10 monclient(hunting): _negotiate 2023-10-30T12:40:36.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.292+0000 7f0613fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:36.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.292+0000 7f0613fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:36.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.292+0000 7f0613fff640 10 monclient(hunting): my global_id is 22159 2023-10-30T12:40:36.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.292+0000 7f0613fff640 1 -- 172.21.15.148:0/2913076767 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05fc003520 con 0x7f06140b7c10 2023-10-30T12:40:36.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.293+0000 7f0618e6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:36.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.293+0000 7f0618e6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:36.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.293+0000 7f0618e6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:36.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.294+0000 7f0618e6b640 10 monclient(hunting): handle_auth_done global_id 16335 payload 274 2023-10-30T12:40:36.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.294+0000 7f0618e6b640 1 -- 172.21.15.148:0/2913076767 >> v1:172.21.15.150:6789/0 conn(0x7f06140b7c10 legacy=0x7f06140a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.295+0000 7f0618e6b640 1 -- 172.21.15.148:0/2913076767 >> v1:172.21.15.148:6789/0 conn(0x7f061413e730 legacy=0x7f061413eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.295+0000 7f0618e6b640 10 monclient: _finish_hunting 0 2023-10-30T12:40:36.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.295+0000 7f0618e6b640 1 monclient: found mon.noname-a 2023-10-30T12:40:36.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.295+0000 7f0618e6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:40:36.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.295+0000 7f0618e6b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:40:36.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.295+0000 7f0618e6b640 1 -- 172.21.15.148:0/2913076767 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06140aa910 con 0x7f0614141940 2023-10-30T12:40:36.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.297+0000 7f0618e6b640 1 --2- 172.21.15.148:0/2913076767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 0x7f0614141d90 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7f06080097a0 tx=0x7f060802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b4ef6188cdd59d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 1 -- 172.21.15.148:0/2913076767 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0608031830 con 0x7f0614141940 2023-10-30T12:40:36.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:36.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:40:36.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 10 monclient: dump: 2023-10-30T12:40:36.304 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:36.304 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:36.304 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:36.304 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:36.305 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:36.305 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:36.305 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:36.305 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:36.305 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:36.306 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:36.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 10 monclient: _finish_auth 0 2023-10-30T12:40:36.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 10 monclient: _check_auth_tickets 2023-10-30T12:40:36.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:40:36.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 1 -- 172.21.15.148:0/2913076767 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060802d040 con 0x7f0614141940 2023-10-30T12:40:36.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f061b8d0640 5 monclient: authenticate success, global_id 16335 2023-10-30T12:40:36.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f061b8d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:40:36.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:40:36.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f061b8d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:40:36.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 1 -- 172.21.15.148:0/2913076767 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0608031bc0 con 0x7f0614141940 2023-10-30T12:40:36.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f061b8d0640 4 set_mon_vals no callback set 2023-10-30T12:40:36.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:36.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:40:36.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.301+0000 7f0613fff640 10 monclient: dump: 2023-10-30T12:40:36.308 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:36.308 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:36.309 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:36.309 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:36.309 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:36.309 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:36.309 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:36.310 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:36.310 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:36.310 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:36.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.302+0000 7f061b8d0640 10 monclient: shutdown 2023-10-30T12:40:36.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.302+0000 7f061b8d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:40:36.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.302+0000 7f061b8d0640 1 -- 172.21.15.148:0/2913076767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 msgr2=0x7f0614141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.302+0000 7f061b8d0640 1 --2- 172.21.15.148:0/2913076767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 0x7f0614141d90 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7f06080097a0 tx=0x7f060802b8d0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.302+0000 7f061b8d0640 1 -- 172.21.15.148:0/2913076767 shutdown_connections 2023-10-30T12:40:36.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.302+0000 7f061b8d0640 1 --2- 172.21.15.148:0/2913076767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 0x7f0614141d90 unknown :-1 s=CLOSED pgs=3246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.302+0000 7f061b8d0640 1 -- 172.21.15.148:0/2913076767 >> 172.21.15.148:0/2913076767 conn(0x7f061406c660 msgr2=0x7f06140b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:36.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.302+0000 7f061b8d0640 1 -- 172.21.15.148:0/2913076767 shutdown_connections 2023-10-30T12:40:36.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.302+0000 7f061b8d0640 1 -- 172.21.15.148:0/2913076767 wait complete. 2023-10-30T12:40:36.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.303+0000 7f061b8d0640 10 monclient: build_initial_monmap 2023-10-30T12:40:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.303+0000 7f061b8d0640 10 monclient: monmap: 2023-10-30T12:40:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2023-10-30T12:40:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:40:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2023-10-30T12:40:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2023-10-30T12:40:36.317 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2023-10-30T12:40:36.317 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2023-10-30T12:40:36.317 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.noname-a 2023-10-30T12:40:36.317 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2023-10-30T12:40:36.317 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.noname-c 2023-10-30T12:40:36.317 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:36.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.303+0000 7f061b8d0640 1 librados: starting msgr at 2023-10-30T12:40:36.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.303+0000 7f061b8d0640 1 librados: starting objecter 2023-10-30T12:40:36.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.303+0000 7f061b8d0640 1 Processor -- start 2023-10-30T12:40:36.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 1 -- start start 2023-10-30T12:40:36.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 1 librados: setting wanted keys 2023-10-30T12:40:36.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 1 librados: calling monclient init 2023-10-30T12:40:36.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient: init 2023-10-30T12:40:36.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:40:36.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:40:36.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06140b7c10 0x7f06141383a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient(hunting): picked mon.noname-b con 0x7f06140b7c10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-30T12:40:36.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f061413e730 0x7f06140acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient(hunting): picked mon.noname-c con 0x7f061413e730 addr [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] 2023-10-30T12:40:36.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 0x7f0614136420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient(hunting): picked mon.noname-a con 0x7f0614141940 addr [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2023-10-30T12:40:36.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:36.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061413eb60 con 0x7f0614141940 2023-10-30T12:40:36.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:36.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06140ad330 con 0x7f06140b7c10 2023-10-30T12:40:36.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:40:36.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06140ac780 con 0x7f061413e730 2023-10-30T12:40:36.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:40:36.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.304+0000 7f061b8d0640 10 monclient(hunting): authenticate will time out at 2928.509467s 2023-10-30T12:40:36.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0619e6d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 0x7f0614136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0618e6b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f061413e730 0x7f06140acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f061966c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06140b7c10 0x7f06141383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0619e6d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 0x7f0614136420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43784/0 (socket says 172.21.15.148:43784) 2023-10-30T12:40:36.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0619e6d640 1 -- 172.21.15.148:0/3809251388 learned_addr learned my addr 172.21.15.148:0/3809251388 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T12:40:36.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0619e6d640 10 monclient(hunting): get_auth_request con 0x7f0614141940 auth_method 0 2023-10-30T12:40:36.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0619e6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:36.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0619e6d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:36.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0619e6d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:36.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f061966c640 10 monclient(hunting): get_auth_request con 0x7f06140b7c10 auth_method 0 2023-10-30T12:40:36.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f061966c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:36.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f061966c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:36.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f061966c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:36.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0618e6b640 10 monclient(hunting): get_auth_request con 0x7f061413e730 auth_method 0 2023-10-30T12:40:36.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0618e6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:40:36.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0618e6b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:40:36.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0618e6b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:40:36.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0619e6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:36.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0619e6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:36.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f0619e6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:36.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f061966c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:36.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f061966c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:36.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.305+0000 7f061966c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:36.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.306+0000 7f0618e6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:40:36.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.306+0000 7f0618e6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:40:36.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.306+0000 7f0618e6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:40:36.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f0619e6d640 10 monclient(hunting): handle_auth_done global_id 16341 payload 824 2023-10-30T12:40:36.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f0619e6d640 1 -- 172.21.15.148:0/3809251388 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f061413e730 msgr2=0x7f06140acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f0619e6d640 1 --2- 172.21.15.148:0/3809251388 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f061413e730 0x7f06140acdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f0619e6d640 1 -- 172.21.15.148:0/3809251388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06140b7c10 msgr2=0x7f06141383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f0619e6d640 1 --2- 172.21.15.148:0/3809251388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06140b7c10 0x7f06141383a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f0619e6d640 10 monclient: _finish_hunting 0 2023-10-30T12:40:36.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f0619e6d640 1 monclient: found mon.noname-a 2023-10-30T12:40:36.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f0619e6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:40:36.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f0619e6d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.148:3300/0 2023-10-30T12:40:36.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f0619e6d640 1 -- 172.21.15.148:0/3809251388 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06140ac950 con 0x7f0614141940 2023-10-30T12:40:36.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.308+0000 7f061966c640 1 --2- 172.21.15.148:0/3809251388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06140b7c10 0x7f06141383a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:40:36.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.310+0000 7f0619e6d640 1 --2- 172.21.15.148:0/3809251388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 0x7f0614136420 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f060c00c3c0 tx=0x7f060c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.310+0000 7f0611ffb640 10 client.?.objecter ms_handle_connect 0x7f0614141940 2023-10-30T12:40:36.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.310+0000 7f0611ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:40:36.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 1 -- 172.21.15.148:0/3809251388 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c0052d0 con 0x7f0614141940 2023-10-30T12:40:36.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:40:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 10 monclient: dump: 2023-10-30T12:40:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:36.332 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:36.332 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:36.332 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:36.332 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:36.332 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:36.333 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:36.333 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:36.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 10 monclient: _finish_auth 0 2023-10-30T12:40:36.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:40:36.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:40:36.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 1 -- 172.21.15.148:0/3809251388 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060c00e040 con 0x7f0614141940 2023-10-30T12:40:36.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:40:36.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 1 -- 172.21.15.148:0/3809251388 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c007260 con 0x7f0614141940 2023-10-30T12:40:36.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f061b8d0640 5 monclient: authenticate success, global_id 16341 2023-10-30T12:40:36.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f061b8d0640 10 monclient: _renew_subs 2023-10-30T12:40:36.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f061b8d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:40:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f061b8d0640 1 -- 172.21.15.148:0/3809251388 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0614136960 con 0x7f0614141940 2023-10-30T12:40:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:40:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:40:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f0611ffb640 10 monclient: dump: 2023-10-30T12:40:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2023-10-30T12:40:36.336 INFO:teuthology.orchestra.run.smithi148.stderr:fsid e2fdab2f-1adf-4b35-9eee-e40114aa8bbb 2023-10-30T12:40:36.336 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:36.336 INFO:teuthology.orchestra.run.smithi148.stderr:created 2023-10-30T12:16:11.988849+0000 2023-10-30T12:40:36.336 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 18 (reef) 2023-10-30T12:40:36.336 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 3 2023-10-30T12:40:36.337 INFO:teuthology.orchestra.run.smithi148.stderr:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T12:40:36.337 INFO:teuthology.orchestra.run.smithi148.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T12:40:36.337 INFO:teuthology.orchestra.run.smithi148.stderr:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2023-10-30T12:40:36.337 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:40:36.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f06137fe640 4 set_mon_vals no callback set 2023-10-30T12:40:36.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f061b8d0640 10 client.16341.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:40:36.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f061b8d0640 10 monclient: _renew_subs 2023-10-30T12:40:36.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f061b8d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:40:36.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f061b8d0640 1 -- 172.21.15.148:0/3809251388 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0614136e50 con 0x7f0614141940 2023-10-30T12:40:36.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.314+0000 7f061b8d0640 1 librados: init done 2023-10-30T12:40:36.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.315+0000 7f061b8d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:40:36.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.315+0000 7f061b8d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:40:36.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.315+0000 7f061b8d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:40:36.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.315+0000 7f061b8d0640 1 -- 172.21.15.148:0/3809251388 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06140aa350 con 0x7f0614141940 2023-10-30T12:40:36.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.334+0000 7f0611ffb640 1 -- 172.21.15.148:0/3809251388 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f060c0073d0 con 0x7f0614141940 2023-10-30T12:40:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.334+0000 7f0611ffb640 1 --2- 172.21.15.148:0/3809251388 >> v2:172.21.15.148:6800/3205090179 conn(0x7f060006c810 0x7f060006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.334+0000 7f061966c640 1 --2- 172.21.15.148:0/3809251388 >> v2:172.21.15.148:6800/3205090179 conn(0x7f060006c810 0x7f060006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.334+0000 7f061966c640 10 monclient: get_auth_request con 0x7f060006c810 auth_method 0 2023-10-30T12:40:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.335+0000 7f061966c640 1 --2- 172.21.15.148:0/3809251388 >> v2:172.21.15.148:6800/3205090179 conn(0x7f060006c810 0x7f060006ecd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0604001700 tx=0x7f0604001a90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.335+0000 7f0611ffb640 10 client.16341.objecter ms_handle_connect 0x7f060006c810 2023-10-30T12:40:36.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.337+0000 7f0611ffb640 1 -- 172.21.15.148:0/3809251388 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(900..900 src has 739..900) v4 ==== 5133+0+0 (secure 0 0 0) 0x7f060c057940 con 0x7f0614141940 2023-10-30T12:40:36.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.337+0000 7f0611ffb640 10 client.16341.objecter ms_dispatch 0x7f0614000b90 osd_map(900..900 src has 739..900) v4 2023-10-30T12:40:36.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.337+0000 7f0611ffb640 3 client.16341.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:40:36.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.337+0000 7f0611ffb640 3 client.16341.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:40:36.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.337+0000 7f0611ffb640 20 client.16341.objecter dump_active .. 0 homeless 2023-10-30T12:40:36.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.572+0000 7f0611ffb640 1 -- 172.21.15.148:0/3809251388 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f060c05b9e0 con 0x7f0614141940 2023-10-30T12:40:36.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.572+0000 7f0611ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:40:36.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.572+0000 7f0611ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:40:36.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.751+0000 7f061b8d0640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T12:40:36.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.751+0000 7f061b8d0640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T12:40:36.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.751+0000 7f061b8d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.148:3300/0 2023-10-30T12:40:36.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.751+0000 7f061b8d0640 1 -- 172.21.15.148:0/3809251388 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f06140ac950 con 0x7f0614141940 2023-10-30T12:40:36.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.863+0000 7f0611ffb640 1 -- 172.21.15.148:0/3809251388 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f060c05fb90 con 0x7f0614141940 2023-10-30T12:40:36.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.863+0000 7f0611ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T12:40:36.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.863+0000 7f0611ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:40:36.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.863+0000 7f0611ffb640 1 -- 172.21.15.148:0/3809251388 <== mon.0 v2:172.21.15.148:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f060c053ab0 con 0x7f0614141940 2023-10-30T12:40:36.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.863+0000 7f0611ffb640 10 monclient: handle_config config(1 keys) v1 2023-10-30T12:40:36.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.863+0000 7f06137fe640 4 set_mon_vals no callback set 2023-10-30T12:40:36.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.863+0000 7f06137fe640 10 set_mon_vals mon_health_to_clog = false 2023-10-30T12:40:36.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.865+0000 7f061b8d0640 10 librados: watch_flush enter 2023-10-30T12:40:36.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 10 librados: watch_flush exit 2023-10-30T12:40:36.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 20 client.16341.objecter shutdown clearing up homeless session... 2023-10-30T12:40:36.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 10 client.16341.objecter successfully canceled tick 2023-10-30T12:40:36.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 -- 172.21.15.148:0/3809251388 >> v2:172.21.15.148:6800/3205090179 conn(0x7f060006c810 msgr2=0x7f060006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 --2- 172.21.15.148:0/3809251388 >> v2:172.21.15.148:6800/3205090179 conn(0x7f060006c810 0x7f060006ecd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0604001700 tx=0x7f0604001a90 comp rx=0 tx=0).stop 2023-10-30T12:40:36.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 10 monclient: shutdown 2023-10-30T12:40:36.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:40:36.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 -- 172.21.15.148:0/3809251388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 msgr2=0x7f0614136420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 --2- 172.21.15.148:0/3809251388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 0x7f0614136420 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f060c00c3c0 tx=0x7f060c00c880 comp rx=0 tx=0).stop 2023-10-30T12:40:36.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 -- 172.21.15.148:0/3809251388 shutdown_connections 2023-10-30T12:40:36.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 --2- 172.21.15.148:0/3809251388 >> v2:172.21.15.148:6800/3205090179 conn(0x7f060006c810 0x7f060006ecd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 --2- 172.21.15.148:0/3809251388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0614141940 0x7f0614136420 unknown :-1 s=CLOSED pgs=3247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 --2- 172.21.15.148:0/3809251388 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f061413e730 0x7f06140acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 --2- 172.21.15.148:0/3809251388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06140b7c10 0x7f06141383a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 -- 172.21.15.148:0/3809251388 >> 172.21.15.148:0/3809251388 conn(0x7f061406c660 msgr2=0x7f06140738f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:36.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 -- 172.21.15.148:0/3809251388 shutdown_connections 2023-10-30T12:40:36.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 -- 172.21.15.148:0/3809251388 wait complete. 2023-10-30T12:40:36.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T12:40:36.866+0000 7f061b8d0640 1 librados: shutdown 2023-10-30T12:40:36.882 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T12:40:36.882 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T12:40:36.883 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T12:40:36.883 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:40:36.897 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:40:36.885+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86478) UID: 0 2023-10-30T12:40:36.898 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T12:40:36.887+0000 b47e640 -1 osd.0 900 *** Got signal Terminated *** 2023-10-30T12:40:37.529 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:52.603 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:37.533 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:52.607 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:37.548 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:52.622 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:37.578 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:52.651 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:37.578 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:52.625 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:37.579 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:52.627 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:37.581 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:52.655 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:37.594 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:52.641 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:37.597 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:52.671 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:38.313 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:53.360 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:38.314 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:53.361 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:38.322 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:23:53.370 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:38.394 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:53.468 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:38.395 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:53.469 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:38.405 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:23:53.478 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:38.429 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:53.503 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:38.430 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:53.504 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:38.439 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:23:53.513 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:52.593 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:07.667 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:52.600 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:07.673 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:52.613 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:07.686 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:52.651 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:07.697 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:52.651 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:07.699 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:52.652 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:07.724 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:52.652 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:07.725 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:52.665 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:07.712 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:52.667 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:07.741 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:55.854 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:10.928 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:55.857 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:10.931 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:55.871 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:10.944 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:55.907 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:10.954 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:55.908 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:10.955 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:55.921 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:10.968 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:55.923 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:10.997 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:55.927 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:11.001 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:55.940 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:11.014 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:57.602 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:12.676 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:57.606 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:12.680 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:57.620 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:12.693 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:57.652 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:12.699 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:57.655 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:12.702 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:57.665 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:12.712 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:57.675 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:12.748 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:57.681 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:12.755 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:57.692 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:12.766 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:17.671 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:32.745 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:17.674 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:32.748 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:17.698 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:32.771 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:17.722 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:32.795 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:17.725 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:32.773 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:17.726 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:32.799 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:17.727 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:32.774 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:17.745 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:32.792 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:17.748 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:32.821 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:18.282 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:33.329 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:18.283 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:33.330 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:18.292 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:33.340 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:18.334 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:33.405 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:18.335 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:33.406 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:18.340 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:33.414 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:18.401 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:33.474 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:18.402 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:33.476 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:18.410 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:33.484 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:29.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-30T12:41:31.796 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T12:41:31.796 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T12:41:31.797 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:41:31.809 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T12:41:31.799+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86488) UID: 0 2023-10-30T12:41:31.809 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T12:41:31.800+0000 b47e640 -1 osd.1 902 *** Got signal Terminated *** 2023-10-30T12:41:35.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-30T12:41:35.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-30T12:41:37.831 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:52.905 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:37.839 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:52.912 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:37.860 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:24:52.934 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:37.887 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:52.934 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:37.887 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:52.961 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:37.890 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:52.964 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:37.891 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:52.938 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:37.898 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T12:41:37.899 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T12:41:37.899 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:41:37.909 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:24:52.956 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:37.910 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:24:52.984 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:37.912 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:41:37.902+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86490) UID: 0 2023-10-30T12:41:37.912 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T12:41:37.903+0000 b47e640 -1 osd.2 906 *** Got signal Terminated *** 2023-10-30T12:41:41.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-30T12:41:57.801 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:12.874 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:57.805 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:12.878 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:57.824 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:12.898 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:57.853 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:12.927 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:57.854 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:12.901 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:57.856 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:12.904 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:57.860 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:12.933 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:57.876 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:12.924 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:57.887 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:12.961 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:58.438 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:13.485 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:58.439 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:13.486 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:58.447 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:13.494 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:58.488 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:13.562 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:58.489 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:13.563 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:58.498 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:13.572 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:58.521 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:13.595 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:58.523 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:13.597 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:58.534 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:13.608 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:17.870 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:32.943 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:17.931 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:33.004 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:17.971 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:33.018 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:17.975 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:33.022 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:17.977 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:33.051 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:17.980 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:33.054 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:17.983 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:33.057 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.024 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:33.071 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.036 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:33.110 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.583 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:33.630 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.583 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:33.631 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.596 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:33.643 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.678 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:33.751 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.680 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:33.754 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.690 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:33.763 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.692 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:33.766 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.693 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:33.767 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:18.705 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:33.779 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:37.951 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:53.024 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:37.979 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:53.052 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:37.981 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:53.054 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:37.981 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:53.055 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:37.999 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:53.072 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:38.000 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:53.074 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:38.006 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:53.053 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:38.007 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:53.055 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:38.023 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:53.070 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:42.936 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:58.009 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:42.941 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:58.015 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:42.953 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:25:58.027 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:42.987 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:58.061 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:42.992 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:58.066 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:42.994 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:58.041 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:42.995 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:58.042 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:43.004 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:25:58.077 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:43.006 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:25:58.053 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:57.935 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:13.008 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:57.985 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:13.059 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:57.989 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:13.036 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:57.993 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:13.040 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:58.002 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:13.076 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:58.013 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:13.060 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:58.046 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:13.120 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:58.050 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:13.124 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:58.067 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:13.140 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:02.996 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:18.069 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:02.997 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:18.071 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:03.006 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:18.080 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:03.042 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:18.115 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:03.045 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:18.119 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:03.045 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:18.092 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:03.046 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:18.094 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:03.057 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:18.104 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:03.062 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:18.136 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:18.031 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:33.104 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:18.034 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:33.108 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:18.051 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:33.125 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:18.084 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:33.130 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:18.084 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:33.157 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:18.087 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:33.134 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:18.091 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:33.165 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:18.103 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:33.150 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:18.113 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:33.186 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:23.045 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:38.118 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:23.050 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:38.121 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:23.064 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:38.137 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:23.067 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:38.115 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:23.071 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:38.119 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:23.083 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:38.130 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:23.097 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:38.171 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:23.099 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:38.173 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:23.113 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:38.186 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:38.089 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:53.163 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:38.091 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:53.165 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:38.105 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:53.179 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:38.141 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:53.188 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:38.142 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:53.216 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:38.143 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:53.190 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:38.144 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:53.218 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:38.157 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:53.204 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:38.162 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:53.236 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:43.055 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:58.129 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:43.104 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:58.151 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:43.104 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:58.178 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:43.105 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:58.179 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:43.107 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:58.154 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:43.108 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:58.182 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:43.120 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:26:58.167 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:43.121 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:26:58.195 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:43.125 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:26:58.198 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:46.027 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T12:43:46.027 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T12:43:46.028 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:43:46.045 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T12:43:46.035+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86492) UID: 0 2023-10-30T12:43:46.045 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T12:43:46.037+0000 b47e640 -1 osd.3 908 *** Got signal Terminated *** 2023-10-30T12:43:58.154 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:13.228 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:58.159 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:13.233 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:58.184 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:13.258 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:58.208 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:13.255 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:58.211 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:13.258 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:58.215 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:13.289 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:58.217 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:13.291 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:58.228 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:13.275 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:43:58.234 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:13.308 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.168 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:18.241 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.173 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:18.246 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.185 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:18.259 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.219 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:18.293 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.222 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:18.270 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.224 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:18.271 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.225 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:18.299 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.234 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:18.281 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.235 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:18.309 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.861 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:18.908 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.862 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:18.909 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.874 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:18.921 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.931 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:19.005 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.932 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:19.006 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.944 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:19.018 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.957 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:19.031 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.958 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:19.032 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:03.971 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:19.045 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:18.163 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:33.237 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:18.213 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:33.286 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:18.214 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:33.262 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:18.218 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:33.265 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:18.235 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:33.308 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:18.236 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:33.283 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:18.281 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:33.355 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:18.286 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:33.359 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:18.301 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:33.375 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:23.223 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:38.297 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:23.225 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:38.299 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:23.238 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:38.312 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:23.273 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:38.347 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:23.275 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:38.349 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:23.277 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:38.325 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:23.280 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:38.327 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:23.291 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:38.365 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:23.292 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:38.339 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:38.216 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:53.289 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:38.263 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:53.337 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:38.264 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:53.338 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:38.265 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:53.339 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:38.284 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:53.357 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:38.284 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:53.357 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:38.314 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:53.362 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:38.317 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:53.364 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:38.332 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:53.379 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:40.939 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T12:44:40.939 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T12:44:40.940 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:44:40.952 INFO:tasks.ceph.osd.4.smithi150.stderr:2023-10-30T12:44:40.942+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86360) UID: 0 2023-10-30T12:44:40.955 INFO:tasks.ceph.osd.4.smithi150.stderr:2023-10-30T12:44:40.944+0000 b47e640 -1 osd.4 910 *** Got signal Terminated *** 2023-10-30T12:44:43.273 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:58.347 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:43.278 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:58.351 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:43.291 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:27:58.365 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:43.324 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:58.371 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:43.327 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:58.400 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:43.328 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:58.375 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:43.330 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:58.404 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:43.339 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:27:58.386 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:43.346 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:27:58.420 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:47.041 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T12:44:47.042 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T12:44:47.042 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:44:47.055 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T12:44:47.045+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86363) UID: 0 2023-10-30T12:44:47.055 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T12:44:47.046+0000 b47e640 -1 osd.5 912 *** Got signal Terminated *** 2023-10-30T12:44:53.312 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:08.386 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:53.317 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:08.390 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:53.334 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:08.407 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:53.362 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:08.409 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:53.363 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:08.410 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:53.366 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:08.439 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:53.371 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:08.444 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:53.378 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:08.426 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:44:53.384 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:08.458 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:03.337 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:18.410 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:03.341 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:18.414 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:03.356 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:18.430 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:03.388 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:18.435 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:03.391 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:18.438 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:03.392 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:18.465 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:03.394 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:18.468 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:03.403 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:18.450 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:03.406 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:18.479 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:18.397 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:33.471 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:18.404 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:33.477 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:18.421 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:33.495 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:18.448 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:33.495 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:18.450 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:33.497 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:18.451 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:33.525 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:18.453 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:33.527 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:18.466 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:33.513 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:18.477 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:33.551 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:23.418 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:38.491 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:23.424 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:38.497 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:23.456 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:38.530 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:23.466 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:38.514 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:23.469 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:38.516 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:23.475 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:38.549 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:23.478 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:38.552 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:23.479 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:38.526 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:23.489 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:38.563 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:38.460 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:53.533 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:38.465 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:53.538 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:38.483 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:53.557 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:38.525 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:53.572 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:38.528 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:53.575 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:38.539 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:53.612 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:38.542 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:53.616 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:38.549 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:53.596 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:38.557 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:53.631 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:43.484 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:58.558 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:43.486 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:58.560 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:43.497 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:28:58.571 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:43.535 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:58.582 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:43.536 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:58.583 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:43.537 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:58.611 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:43.539 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:58.613 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:43.545 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:28:58.592 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:43.553 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:28:58.627 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:58.479 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:13.552 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:58.527 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:13.601 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:58.545 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:13.618 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:58.575 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:13.622 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:58.578 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:13.652 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:58.578 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:13.626 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:58.582 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:13.656 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:58.595 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:13.642 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:45:58.600 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:13.674 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:03.539 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:18.612 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:03.543 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:18.617 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:03.559 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:18.633 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:03.585 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:18.632 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:03.588 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:18.635 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:03.591 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:18.665 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:03.598 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:18.672 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:03.599 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:18.646 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:03.609 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:18.683 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:16.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-10-30T12:46:18.565 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T12:46:18.565 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-30T12:46:18.566 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:46:18.577 INFO:tasks.ceph.osd.6.smithi150.stderr:2023-10-30T12:46:18.568+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86366) UID: 0 2023-10-30T12:46:18.581 INFO:tasks.ceph.osd.6.smithi150.stderr:2023-10-30T12:46:18.570+0000 b47e640 -1 osd.6 914 *** Got signal Terminated *** 2023-10-30T12:46:18.586 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:33.659 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:18.588 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:33.662 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:18.603 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:33.677 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:18.636 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:33.710 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:18.638 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:33.685 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:18.640 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:33.714 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:18.644 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:33.691 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:18.657 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:33.731 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:18.661 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:33.708 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:21.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-10-30T12:46:21.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-10-30T12:46:23.696 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:38.770 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:23.698 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:38.772 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:23.708 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:38.782 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:23.740 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:38.787 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:23.743 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:38.790 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:23.748 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:38.822 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:23.750 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:38.823 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:23.753 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:38.801 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:23.760 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:38.834 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:24.667 INFO:tasks.ceph.osd.6:Stopped 2023-10-30T12:46:24.668 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-30T12:46:24.668 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:46:24.680 INFO:tasks.ceph.osd.7.smithi150.stderr:2023-10-30T12:46:24.670+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86367) UID: 0 2023-10-30T12:46:24.682 INFO:tasks.ceph.osd.7.smithi150.stderr:2023-10-30T12:46:24.672+0000 b47e640 -1 osd.7 914 *** Got signal Terminated *** 2023-10-30T12:46:27.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-10-30T12:46:38.693 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:53.766 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:38.695 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:53.769 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:38.712 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:29:53.786 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:38.739 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:53.786 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:38.743 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:53.817 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:38.744 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:53.791 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:38.748 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:53.822 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:38.760 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:29:53.807 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:38.767 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:29:53.841 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:48.717 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:30:03.791 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:48.720 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:30:03.794 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:48.735 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:30:03.809 86143== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:48.765 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:30:03.813 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:48.767 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:30:03.841 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:48.768 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:30:03.816 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:48.772 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:30:03.846 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:48.781 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:30:03.828 85836== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:48.787 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:30:03.861 86149== Warning: unimplemented fcntl command: 1036 2023-10-30T12:46:55.176 INFO:tasks.ceph.osd.7:Stopped 2023-10-30T12:46:55.177 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T12:46:55.177 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-30T12:46:55.178 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:46:55.204 INFO:tasks.ceph.mgr.y:Stopped 2023-10-30T12:46:55.204 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T12:46:55.205 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:46:55.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-10-30T12:46:55.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-10-30T12:46:55.231 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T12:46:55.231 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T12:46:55.232 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:46:55.232 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:46:55.258 INFO:tasks.ceph.mon.a.smithi148.stderr:2023-10-30T12:46:55.247+0000 9f5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86115) UID: 0 2023-10-30T12:46:55.259 INFO:tasks.ceph.mon.a.smithi148.stderr:2023-10-30T12:46:55.248+0000 9f5f640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-30T12:46:55.884 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:30:10.958 86143== 2023-10-30T12:46:55.885 INFO:tasks.ceph.mon.a.smithi148.stderr:==00:00:30:10.958 86143== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T12:46:55.901 INFO:tasks.ceph.mon.a.smithi148.stderr:daemon-helper: command failed with exit status 42 2023-10-30T12:47:00.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-30T12:47:00.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-10-30T12:47:00.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-10-30T12:47:01.334 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T12:47:01.335 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-10-30T12:47:01.335 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:47:01.336 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T12:47:01.336 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:47:01.351 INFO:tasks.ceph.mon.c.smithi148.stderr:2023-10-30T12:47:01.339+0000 9f5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86117) UID: 0 2023-10-30T12:47:01.352 INFO:tasks.ceph.mon.c.smithi148.stderr:2023-10-30T12:47:01.340+0000 9f5f640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-10-30T12:47:02.085 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:30:17.158 86149== 2023-10-30T12:47:02.085 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:30:17.158 86149== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T12:47:02.099 INFO:tasks.ceph.mon.c.smithi148.stderr:daemon-helper: command failed with exit status 42 2023-10-30T12:47:05.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-10-30T12:47:05.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-30T12:47:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T12:47:07.439 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-10-30T12:47:07.439 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T12:47:07.440 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T12:47:07.440 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:47:07.455 INFO:tasks.ceph.mon.b.smithi150.stderr:2023-10-30T12:47:07.443+0000 9f5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 85832) UID: 0 2023-10-30T12:47:07.456 INFO:tasks.ceph.mon.b.smithi150.stderr:2023-10-30T12:47:07.444+0000 9f5f640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-10-30T12:47:08.215 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:30:23.262 85836== 2023-10-30T12:47:08.215 INFO:tasks.ceph.mon.b.smithi150.stderr:==00:00:30:23.262 85836== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T12:47:08.229 INFO:tasks.ceph.mon.b.smithi150.stderr:daemon-helper: command failed with exit status 42 2023-10-30T12:47:10.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-10-30T12:47:10.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-10-30T12:47:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T12:47:13.543 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-10-30T12:47:13.544 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T12:47:13.544 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T12:47:13.544 DEBUG:teuthology.orchestra.run.smithi148:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-10-30T12:47:13.683 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:47:13.684 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T12:47:13.783 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:47:13.783 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T12:47:13.849 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:47:13.850 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T12:47:13.909 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:47:13.910 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T12:47:13.965 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:47:13.965 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T12:47:14.140 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:47:14.141 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T12:47:14.204 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:47:14.205 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-30T12:47:14.255 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:47:14.255 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-30T12:47:14.313 INFO:tasks.ceph:Archiving mon data... 2023-10-30T12:47:14.315 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076/data/mon.a.tgz 2023-10-30T12:47:14.315 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2023-10-30T12:47:14.332 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.aF4w8VnIEY 2023-10-30T12:47:14.333 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.aF4w8VnIEY 2023-10-30T12:47:14.498 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.aF4w8VnIEY 2023-10-30T12:47:14.582 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.aF4w8VnIEY is 468KB 2023-10-30T12:47:14.650 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.aF4w8VnIEY 2023-10-30T12:47:14.664 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076/data/mon.c.tgz 2023-10-30T12:47:14.665 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2023-10-30T12:47:14.718 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.52rmHDVeNd 2023-10-30T12:47:14.719 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.52rmHDVeNd 2023-10-30T12:47:14.866 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.52rmHDVeNd 2023-10-30T12:47:14.950 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.52rmHDVeNd is 469KB 2023-10-30T12:47:15.047 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.52rmHDVeNd 2023-10-30T12:47:15.061 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076/data/mon.b.tgz 2023-10-30T12:47:15.062 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2023-10-30T12:47:15.080 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.VVMmKrC903 2023-10-30T12:47:15.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.VVMmKrC903 2023-10-30T12:47:15.249 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.VVMmKrC903 2023-10-30T12:47:15.334 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.VVMmKrC903 is 469KB 2023-10-30T12:47:15.431 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.VVMmKrC903 2023-10-30T12:47:15.445 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T12:47:15.445 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T12:47:15.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T12:47:15.510 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-30T12:47:15.511 DEBUG:teuthology.orchestra.run.smithi148:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T12:47:15.515 DEBUG:teuthology.orchestra.run.smithi150:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T12:47:15.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-10-30T12:47:15.554 INFO:teuthology.orchestra.run.smithi148.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2023-10-30T12:47:15.554 INFO:teuthology.orchestra.run.smithi148.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2023-10-30T12:47:15.554 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2023-10-30T12:47:15.555 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2023-10-30T12:47:15.555 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2023-10-30T12:47:15.555 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2023-10-30T12:47:15.589 INFO:teuthology.orchestra.run.smithi150.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2023-10-30T12:47:15.590 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2023-10-30T12:47:15.590 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2023-10-30T12:47:15.590 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T12:47:15.593 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076/remote/smithi148/crash 2023-10-30T12:47:15.594 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T12:47:15.621 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076/remote/smithi150/crash 2023-10-30T12:47:15.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T12:47:15.656 INFO:tasks.ceph:Compressing logs... 2023-10-30T12:47:15.656 DEBUG:teuthology.orchestra.run.smithi148:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T12:47:15.665 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T12:47:15.688 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158598.log 2023-10-30T12:47:15.690 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92575.log 2023-10-30T12:47:15.690 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86565.log 2023-10-30T12:47:15.690 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158598.log: /var/log/ceph/ceph-client.admin.92575.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T12:47:15.690 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158598.log.gz 2023-10-30T12:47:15.691 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.86565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91266.log 2023-10-30T12:47:15.691 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116440.log 2023-10-30T12:47:15.714 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90537.log 2023-10-30T12:47:15.714 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90840.log 2023-10-30T12:47:15.714 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mon.a.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.86565.log.gz 2023-10-30T12:47:15.714 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91266.log: /var/log/ceph/ceph-client.admin.116440.log: /var/log/ceph/ceph-client.admin.90537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89744.log 2023-10-30T12:47:15.715 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5/var/log/ceph/ceph-client.admin.90840.log: --verbose -- /var/log/ceph/ceph-client.admin.91987.log 2023-10-30T12:47:15.715 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% 87.8%/var/log/ceph/ceph-client.admin.89744.log: -- replaced with /var/log/ceph/ceph-client.admin.90537.log.gz 2023-10-30T12:47:15.715 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91266.log.gz 2023-10-30T12:47:15.715 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91543.log 2023-10-30T12:47:15.715 INFO:teuthology.orchestra.run.smithi148.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90840.log.gz 2023-10-30T12:47:15.716 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124652.log 2023-10-30T12:47:15.716 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89744.log.gz 2023-10-30T12:47:15.716 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90540.log 2023-10-30T12:47:15.716 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.124652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124651.log 2023-10-30T12:47:15.716 INFO:teuthology.orchestra.run.smithi148.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91543.log.gzgzip -5 --verbose/var/log/ceph/ceph-client.admin.90540.log: -- /var/log/ceph/ceph-client.admin.92198.log 2023-10-30T12:47:15.716 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:47:15.717 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158768.log 2023-10-30T12:47:15.717 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.124651.log: 87.7%/var/log/ceph/ceph-client.admin.92198.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.90540.log.gz 2023-10-30T12:47:15.717 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.116440.log.gz 2023-10-30T12:47:15.717 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151831.log 2023-10-30T12:47:15.717 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5/var/log/ceph/ceph-client.admin.158768.log: --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-10-30T12:47:15.718 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.151831.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-10-30T12:47:15.718 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158768.log.gz 2023-10-30T12:47:15.718 INFO:teuthology.orchestra.run.smithi148.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.91987.log.gz 2023-10-30T12:47:15.718 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/valgrind/mon.c.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.124652.log.gz 2023-10-30T12:47:15.718 INFO:teuthology.orchestra.run.smithi148.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-10-30T12:47:15.719 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-10-30T12:47:15.719 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-10-30T12:47:15.719 INFO:teuthology.orchestra.run.smithi148.stderr: 64.0%/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-10-30T12:47:15.719 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-10-30T12:47:15.719 INFO:teuthology.orchestra.run.smithi148.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-10-30T12:47:15.719 INFO:teuthology.orchestra.run.smithi148.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.124651.log.gz 2023-10-30T12:47:15.720 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-10-30T12:47:15.720 INFO:teuthology.orchestra.run.smithi148.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-10-30T12:47:15.720 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91627.log 2023-10-30T12:47:15.720 INFO:teuthology.orchestra.run.smithi148.stderr: 74.7%/var/log/ceph/valgrind/osd.3.log: -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-10-30T12:47:15.720 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91181.log 2023-10-30T12:47:15.721 INFO:teuthology.orchestra.run.smithi148.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-10-30T12:47:15.721 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91627.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.151831.log.gz 2023-10-30T12:47:15.721 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92441.log 2023-10-30T12:47:15.721 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91181.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158136.log 2023-10-30T12:47:15.721 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91627.log.gz 2023-10-30T12:47:15.721 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92441.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91181.log.gz 2023-10-30T12:47:15.722 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92556.log 2023-10-30T12:47:15.722 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92596.log 2023-10-30T12:47:15.722 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86517.log 2023-10-30T12:47:15.722 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158178.log 2023-10-30T12:47:15.722 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.158136.log.gz 2023-10-30T12:47:15.723 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.86517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158851.log 2023-10-30T12:47:15.723 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158178.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86517.log.gz 2023-10-30T12:47:15.723 INFO:teuthology.orchestra.run.smithi148.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.92556.log.gz 2023-10-30T12:47:15.723 INFO:teuthology.orchestra.run.smithi148.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158178.log.gz 2023-10-30T12:47:15.723 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124653.log 2023-10-30T12:47:15.723 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91970.log 2023-10-30T12:47:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158851.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.92198.log.gz 2023-10-30T12:47:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.124653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158556.log 2023-10-30T12:47:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90535.log 2023-10-30T12:47:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158556.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158851.log.gz 2023-10-30T12:47:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91991.log 2023-10-30T12:47:15.725 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158556.log.gz 2023-10-30T12:47:15.725 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158682.log 2023-10-30T12:47:15.725 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91991.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90535.log.gz 2023-10-30T12:47:15.725 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91417.log 2023-10-30T12:47:15.725 INFO:teuthology.orchestra.run.smithi148.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.124653.log.gz 2023-10-30T12:47:15.725 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114162.log 2023-10-30T12:47:15.726 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91417.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.158682.log.gz 2023-10-30T12:47:15.726 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90539.log 2023-10-30T12:47:15.726 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91417.log.gz 2023-10-30T12:47:15.726 INFO:teuthology.orchestra.run.smithi148.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.92596.log.gz 2023-10-30T12:47:15.726 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.114162.log: gzip -5 --verbose 91.9% -- replaced with /var/log/ceph/ceph-client.admin.92441.log.gz 2023-10-30T12:47:15.726 INFO:teuthology.orchestra.run.smithi148.stderr: -- /var/log/ceph/ceph-client.admin.92435.log 2023-10-30T12:47:15.727 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90359.log 2023-10-30T12:47:15.727 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86157.log 2023-10-30T12:47:15.727 INFO:teuthology.orchestra.run.smithi148.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90539.log.gz 2023-10-30T12:47:15.727 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90165.log 2023-10-30T12:47:15.727 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90359.log: /var/log/ceph/ceph-client.admin.86157.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90359.log.gz 2023-10-30T12:47:15.728 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91244.log 2023-10-30T12:47:15.728 INFO:teuthology.orchestra.run.smithi148.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.114162.log.gz 2023-10-30T12:47:15.728 INFO:teuthology.orchestra.run.smithi148.stderr: 92.3%/var/log/ceph/ceph-client.admin.90165.log: -- replaced with /var/log/ceph/ceph-client.admin.86157.log.gz 2023-10-30T12:47:15.728 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91585.log 2023-10-30T12:47:15.728 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90858.log 2023-10-30T12:47:15.728 INFO:teuthology.orchestra.run.smithi148.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.91970.log.gz 2023-10-30T12:47:15.729 INFO:teuthology.orchestra.run.smithi148.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90165.log.gz 2023-10-30T12:47:15.729 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91585.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119276.log 2023-10-30T12:47:15.729 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91244.log.gz 2023-10-30T12:47:15.729 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91920.log 2023-10-30T12:47:15.729 INFO:teuthology.orchestra.run.smithi148.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91585.log.gz 2023-10-30T12:47:15.730 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.119276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90069.log 2023-10-30T12:47:15.730 INFO:teuthology.orchestra.run.smithi148.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T12:47:15.730 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90858.log.gz 2023-10-30T12:47:15.730 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91920.log: /var/log/ceph/ceph-client.admin.90069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92098.log 2023-10-30T12:47:15.730 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.119276.log.gz 2023-10-30T12:47:15.730 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92684.log 2023-10-30T12:47:15.731 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90069.log.gz 2023-10-30T12:47:15.731 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138497.log 2023-10-30T12:47:15.731 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92684.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.92684.log.gz 2023-10-30T12:47:15.731 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158430.log 2023-10-30T12:47:15.731 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.138497.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.92435.log.gz 2023-10-30T12:47:15.743 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-30T12:47:15.744 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86025.log 2023-10-30T12:47:15.744 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86069.log 2023-10-30T12:47:15.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.86025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86189.log 2023-10-30T12:47:15.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.86069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86229.log 2023-10-30T12:47:15.744 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86025.log.gz 2023-10-30T12:47:15.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.86189.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-10-30T12:47:15.745 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86069.log.gz 2023-10-30T12:47:15.745 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-10-30T12:47:15.745 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.86229.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86189.log.gz 2023-10-30T12:47:15.745 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-10-30T12:47:15.745 INFO:teuthology.orchestra.run.smithi150.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-10-30T12:47:15.745 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-10-30T12:47:15.746 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86229.log.gz 2023-10-30T12:47:15.746 INFO:teuthology.orchestra.run.smithi150.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-10-30T12:47:15.746 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-10-30T12:47:15.746 INFO:teuthology.orchestra.run.smithi150.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-10-30T12:47:15.746 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86109.log 2023-10-30T12:47:15.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/valgrind/mon.b.log: /var/log/ceph/valgrind/osd.4.log: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-10-30T12:47:15.747 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86269.log 2023-10-30T12:47:15.747 INFO:teuthology.orchestra.run.smithi150.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-10-30T12:47:15.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.86109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86149.log 2023-10-30T12:47:15.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.86269.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T12:47:15.748 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86109.log.gz 2023-10-30T12:47:15.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.86149.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T12:47:15.748 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86269.log.gz 2023-10-30T12:47:15.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-30T12:47:15.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.4.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86149.log.gz 2023-10-30T12:47:15.748 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T12:47:15.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T12:47:15.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85973.log 2023-10-30T12:47:15.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T12:47:15.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.85973.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T12:47:15.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85973.log.gz 2023-10-30T12:47:15.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mon.b.log: 92.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T12:47:15.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T12:47:15.757 INFO:teuthology.orchestra.run.smithi150.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T12:47:15.770 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91081.log 2023-10-30T12:47:15.770 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158430.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158430.log.gz 2023-10-30T12:47:15.771 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114163.log 2023-10-30T12:47:15.771 INFO:teuthology.orchestra.run.smithi148.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.138497.log.gz 2023-10-30T12:47:15.771 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90108.log 2023-10-30T12:47:15.771 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91081.log.gz 2023-10-30T12:47:15.771 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.114163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92294.log 2023-10-30T12:47:15.772 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90108.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90108.log.gz 2023-10-30T12:47:15.772 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158052.log 2023-10-30T12:47:15.772 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92294.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.114163.log.gz 2023-10-30T12:47:15.772 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90215.log 2023-10-30T12:47:15.772 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158052.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158052.log.gz 2023-10-30T12:47:15.773 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158724.log 2023-10-30T12:47:15.773 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90215.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90215.log.gz 2023-10-30T12:47:15.773 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91447.log 2023-10-30T12:47:15.773 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158724.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.92294.log.gz 2023-10-30T12:47:15.773 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87436.log 2023-10-30T12:47:15.774 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.158724.log.gz 2023-10-30T12:47:15.774 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138493.log 2023-10-30T12:47:15.774 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.87436.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91447.log.gz 2023-10-30T12:47:15.774 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87436.log.gz 2023-10-30T12:47:15.774 INFO:teuthology.orchestra.run.smithi148.stderr: -- /var/log/ceph/ceph-osd.1.log 2023-10-30T12:47:15.775 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.138493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92016.log 2023-10-30T12:47:15.775 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.1.log: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T12:47:15.775 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92496.log 2023-10-30T12:47:15.775 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92016.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.138493.log.gz 2023-10-30T12:47:15.775 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92726.log 2023-10-30T12:47:15.775 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92496.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.92016.log.gz 2023-10-30T12:47:15.776 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92193.log 2023-10-30T12:47:15.776 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92726.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92726.log.gz 2023-10-30T12:47:15.776 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151833.log 2023-10-30T12:47:15.776 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92193.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92496.log.gz 2023-10-30T12:47:15.776 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116441.log 2023-10-30T12:47:15.777 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.151833.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92575.log.gz 2023-10-30T12:47:15.777 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90532.log 2023-10-30T12:47:15.777 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.116441.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.116441.log.gz 2023-10-30T12:47:15.777 INFO:teuthology.orchestra.run.smithi148.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.151833.log.gz 2023-10-30T12:47:15.777 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157884.log 2023-10-30T12:47:15.778 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138496.log 2023-10-30T12:47:15.778 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.157884.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90532.log.gz 2023-10-30T12:47:15.778 INFO:teuthology.orchestra.run.smithi148.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.157884.log.gz 2023-10-30T12:47:15.778 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91936.log 2023-10-30T12:47:15.778 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.138496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90848.log 2023-10-30T12:47:15.779 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91936.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.138496.log.gz 2023-10-30T12:47:15.779 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92346.log 2023-10-30T12:47:15.779 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90848.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90848.log.gz 2023-10-30T12:47:15.779 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91496.log 2023-10-30T12:47:15.779 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92346.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.91920.log.gz 2023-10-30T12:47:15.779 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90541.log 2023-10-30T12:47:15.780 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91496.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91496.log.gz 2023-10-30T12:47:15.780 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T12:47:15.780 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90541.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90541.log.gz 2023-10-30T12:47:15.780 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157926.log 2023-10-30T12:47:15.780 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.3.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.92346.log.gz 2023-10-30T12:47:15.781 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91161.log 2023-10-30T12:47:15.781 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.157926.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.157926.log.gz 2023-10-30T12:47:15.781 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90533.log 2023-10-30T12:47:15.781 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91161.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91161.log.gz 2023-10-30T12:47:15.781 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91314.log 2023-10-30T12:47:15.781 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90533.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90533.log.gz 2023-10-30T12:47:15.782 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116442.log 2023-10-30T12:47:15.782 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91314.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91314.log.gz 2023-10-30T12:47:15.782 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158220.log 2023-10-30T12:47:15.782 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.116442.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.116442.log.gz 2023-10-30T12:47:15.782 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119274.log 2023-10-30T12:47:15.783 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158220.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.158220.log.gz 2023-10-30T12:47:15.783 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90838.log 2023-10-30T12:47:15.783 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.119274.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.91991.log.gz 2023-10-30T12:47:15.783 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92736.log 2023-10-30T12:47:15.783 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90838.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.119274.log.gz 2023-10-30T12:47:15.784 INFO:teuthology.orchestra.run.smithi148.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90838.log.gz 2023-10-30T12:47:15.784 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91101.log 2023-10-30T12:47:15.784 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158472.log 2023-10-30T12:47:15.784 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91101.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.92736.log.gz 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91101.log.gz 2023-10-30T12:47:15.784 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:47:15.785 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158514.log 2023-10-30T12:47:15.785 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158010.log 2023-10-30T12:47:15.785 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.158472.log.gz 2023-10-30T12:47:15.785 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90523.log 2023-10-30T12:47:15.785 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158010.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.158514.log.gz 2023-10-30T12:47:15.785 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-30T12:47:15.786 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.158010.log.gz 2023-10-30T12:47:15.786 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114164.log 2023-10-30T12:47:15.786 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mgr.y.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90523.log.gz 2023-10-30T12:47:15.786 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92735.log 2023-10-30T12:47:15.789 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.114164.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.114164.log.gz 2023-10-30T12:47:15.789 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90843.log 2023-10-30T12:47:15.789 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92735.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.92735.log.gz 2023-10-30T12:47:15.789 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92725.log 2023-10-30T12:47:15.790 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90843.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90843.log.gz 2023-10-30T12:47:15.790 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158262.log 2023-10-30T12:47:15.790 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92725.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.92725.log.gz 2023-10-30T12:47:15.790 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157968.log 2023-10-30T12:47:15.790 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158262.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158262.log.gz 2023-10-30T12:47:15.791 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90886.log 2023-10-30T12:47:15.791 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.157968.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.157968.log.gz 2023-10-30T12:47:15.791 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92173.log 2023-10-30T12:47:15.791 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90886.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.90886.log.gz 2023-10-30T12:47:15.791 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T12:47:15.803 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151832.log 2023-10-30T12:47:15.807 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.0.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.91936.log.gz 2023-10-30T12:47:15.809 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158304.log 2023-10-30T12:47:15.812 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.151832.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.151832.log.gz 2023-10-30T12:47:15.812 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91959.log 2023-10-30T12:47:15.813 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158304.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158304.log.gz 2023-10-30T12:47:15.814 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158094.log 2023-10-30T12:47:15.821 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91959.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.91959.log.gz 2023-10-30T12:47:15.821 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90841.log 2023-10-30T12:47:15.823 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158094.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158094.log.gz 2023-10-30T12:47:15.823 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90271.log 2023-10-30T12:47:15.824 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90841.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.90841.log.gz 2023-10-30T12:47:15.824 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158388.log 2023-10-30T12:47:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90271.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90271.log.gz 2023-10-30T12:47:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T12:47:15.827 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158388.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.158388.log.gz 2023-10-30T12:47:15.827 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158346.log 2023-10-30T12:47:15.838 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.log: 92.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88384.log 2023-10-30T12:47:15.840 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158346.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158346.log.gz 2023-10-30T12:47:15.840 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T12:47:15.840 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-30T12:47:15.841 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.88384.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88384.log.gz 2023-10-30T12:47:15.842 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80306.log 2023-10-30T12:47:15.855 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158640.log 2023-10-30T12:47:15.855 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.tmp-client.admin.80306.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80306.log.gz 2023-10-30T12:47:15.869 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138494.log 2023-10-30T12:47:15.869 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158640.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158640.log.gz 2023-10-30T12:47:15.870 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91315.log 2023-10-30T12:47:15.877 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.138494.log: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92409.log 2023-10-30T12:47:15.878 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.138494.log.gz 2023-10-30T12:47:15.891 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.91315.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92724.log 2023-10-30T12:47:15.891 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91315.log.gz 2023-10-30T12:47:15.897 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92409.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92409.log.gz 2023-10-30T12:47:15.911 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90317.log 2023-10-30T12:47:15.911 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92724.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.92724.log.gz 2023-10-30T12:47:15.920 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92539.log 2023-10-30T12:47:15.929 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90317.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92236.log 2023-10-30T12:47:15.933 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92539.log: -- replaced with /var/log/ceph/ceph-client.admin.90317.log.gz 2023-10-30T12:47:15.937 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92464.log 2023-10-30T12:47:15.945 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92236.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.92539.log.gz 2023-10-30T12:47:15.946 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92722.log 2023-10-30T12:47:15.958 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92464.log: 91.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138495.log 2023-10-30T12:47:15.958 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92722.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.92236.log.gz 2023-10-30T12:47:15.961 INFO:teuthology.orchestra.run.smithi148.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.admin.92173.log.gz 2023-10-30T12:47:15.966 INFO:teuthology.orchestra.run.smithi148.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.92464.log.gz 2023-10-30T12:47:15.967 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92216.log 2023-10-30T12:47:15.973 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.138495.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.92722.log.gz 2023-10-30T12:47:15.983 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.138495.log.gz 2023-10-30T12:47:15.984 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92011.log 2023-10-30T12:47:16.000 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92216.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.92216.log.gz 2023-10-30T12:47:16.008 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90849.log 2023-10-30T12:47:16.013 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.92011.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92011.log.gz 2023-10-30T12:47:16.021 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86673.log 2023-10-30T12:47:16.022 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.90849.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.90849.log.gz 2023-10-30T12:47:16.030 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158811.log 2023-10-30T12:47:16.041 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.86673.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T12:47:16.041 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.158811.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.158811.log.gz 2023-10-30T12:47:16.044 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86673.log.gz 2023-10-30T12:47:16.109 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mon.c.log: 91.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-30T12:47:16.263 INFO:teuthology.orchestra.run.smithi148.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.92193.log.gz 2023-10-30T12:47:16.803 INFO:teuthology.orchestra.run.smithi148.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92098.log.gz 2023-10-30T12:47:17.588 INFO:teuthology.orchestra.run.smithi150.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T12:47:17.620 INFO:teuthology.orchestra.run.smithi148.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T12:47:18.524 INFO:teuthology.orchestra.run.smithi148.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T12:47:19.184 INFO:teuthology.orchestra.run.smithi150.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-30T12:47:19.314 INFO:teuthology.orchestra.run.smithi150.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T12:47:19.526 INFO:teuthology.orchestra.run.smithi150.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T12:47:19.836 INFO:teuthology.orchestra.run.smithi150.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-30T12:47:19.838 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T12:47:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m4.122s 2023-10-30T12:47:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m16.276s 2023-10-30T12:47:19.839 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.557s 2023-10-30T12:47:19.925 INFO:teuthology.orchestra.run.smithi148.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T12:47:20.008 INFO:teuthology.orchestra.run.smithi148.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T12:47:20.038 INFO:teuthology.orchestra.run.smithi148.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T12:47:20.050 INFO:teuthology.orchestra.run.smithi148.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T12:47:20.052 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T12:47:20.053 INFO:teuthology.orchestra.run.smithi148.stderr:real 0m4.375s 2023-10-30T12:47:20.053 INFO:teuthology.orchestra.run.smithi148.stderr:user 0m22.784s 2023-10-30T12:47:20.053 INFO:teuthology.orchestra.run.smithi148.stderr:sys 0m0.867s 2023-10-30T12:47:20.054 INFO:tasks.ceph:Archiving logs... 2023-10-30T12:47:20.054 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076/remote/smithi148/log 2023-10-30T12:47:20.055 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T12:47:22.985 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076/remote/smithi150/log 2023-10-30T12:47:22.986 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T12:47:25.190 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-10-30T12:47:25.190 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T12:47:25.199 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-10-30T12:47:25.199 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T12:47:25.199 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T12:47:25.202 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T12:47:25.287 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T12:47:25.295 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T12:47:25.296 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:47:25.299 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:47:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2023-10-30T12:47:25.320 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2023-10-30T12:47:25.322 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T12:47:25.322 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2023-10-30T12:47:25.322 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +2184us[+2186us] +/- 87ms 2023-10-30T12:47:25.322 INFO:teuthology.orchestra.run.smithi148.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -1656us[-1654us] +/- 60ms 2023-10-30T12:47:25.322 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +466us[ +466us] +/- 79ms 2023-10-30T12:47:25.323 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:47:25.323 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T12:47:25.323 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-10-30T12:47:25.323 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +2065us[+2082us] +/- 87ms 2023-10-30T12:47:25.323 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -1770us[-1753us] +/- 59ms 2023-10-30T12:47:25.324 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +348us[ +365us] +/- 79ms 2023-10-30T12:47:25.324 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:47:25.324 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T12:47:25.334 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T12:47:25.334 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T12:47:25.342 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-30T12:47:25.460 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-30T12:47:25.558 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T12:47:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:47:25.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T12:47:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:47:25.624 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T12:47:25.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T12:47:25.641 INFO:teuthology.task.internal:Duration was 2977.475120 seconds 2023-10-30T12:47:25.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T12:47:25.650 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T12:47:25.651 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T12:47:25.653 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T12:47:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T12:47:25.708 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T12:47:25.916 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T12:47:25.916 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:47:25.917 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T12:47:25.968 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:47:25.968 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T12:47:25.993 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T12:47:25.994 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T12:47:26.010 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T12:47:26.062 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T12:47:26.062 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T12:47:26.079 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T12:47:26.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T12:47:26.208 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T12:47:26.209 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T12:47:26.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T12:47:26.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T12:47:26.272 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:47:26.277 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:47:26.300 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2023-10-30T12:47:26.327 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2023-10-30T12:47:26.354 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:47:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:47:26.369 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:47:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:47:26.408 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T12:47:26.421 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T12:47:26.421 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076/remote/smithi148 2023-10-30T12:47:26.422 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T12:47:26.458 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076/remote/smithi150 2023-10-30T12:47:26.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T12:47:26.493 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T12:47:26.494 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T12:47:26.496 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T12:47:26.543 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T12:47:26.554 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T12:47:26.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T12:47:26.562 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T12:47:26.563 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T12:47:26.565 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T12:47:26.577 INFO:teuthology.orchestra.run.smithi148.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 12:47 /home/ubuntu/cephtest 2023-10-30T12:47:26.597 INFO:teuthology.orchestra.run.smithi150.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 12:47 /home/ubuntu/cephtest 2023-10-30T12:47:26.598 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T12:47:26.608 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T12:47:26.618 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2023-10-30T12:47:26.619 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T12:47:26.645 DEBUG:teuthology.nuke:shortname: smithi148 2023-10-30T12:47:26.645 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:47:26.670 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:48:37.669932', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErD+fKORMbAS/4SGdN1qk+fU5KuhzMeu5Q6Ym1h8GMQsH5+45LOEyN9dgz8ltlxPMWIuTBvuay8FM/2eK7wFqM='} 2023-10-30T12:47:26.673 DEBUG:teuthology.nuke:shortname: smithi150 2023-10-30T12:47:26.674 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:47:26.695 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441076', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:48:37.667921', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKgJDD52JQV1IuyEaPL+QAJj7lMW3qMUl3NiI2iKvVkDuf8z8E/576IK4HXp620zEqzKzeEFUb7IQIoHyd7zmSY='} 2023-10-30T12:47:26.725 INFO:teuthology.orchestra.console.smithi148:Power off 2023-10-30T12:47:26.726 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T12:47:26.740 INFO:teuthology.orchestra.console.smithi150:Power off 2023-10-30T12:47:26.740 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T12:47:26.749 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2023-10-30T12:47:26.750 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:47:26.762 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2023-10-30T12:47:26.762 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:47:26.772 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-10-30T12:47:26.785 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-10-30T12:47:30.774 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:47:30.786 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:47:30.884 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-10-30T12:47:30.895 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-10-30T12:47:34.886 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:47:34.896 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:47:34.996 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-10-30T12:47:35.004 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2023-10-30T12:47:35.098 INFO:teuthology.orchestra.console.smithi148:Power off completed 2023-10-30T12:47:35.105 INFO:teuthology.orchestra.console.smithi150:Power off completed 2023-10-30T12:47:35.209 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 2977.475119829178 failure_reason: saw valgrind issues flavor: default owner: scheduled_amathuri@teuthology status: fail success: false 2023-10-30T12:47:35.210 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T12:47:35.289 INFO:teuthology.run:FAIL